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California  92152-6800,  (619)  553-0554  or  AUTOVON  553-0554. 
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SUMMARY 


Problem 

Transportable  education  and  training  courseware  is  becoming  increasingly  important  to  the 
military  as  an  alternative  to  residential  classroom  instruction,  which  is  expensive,  usually 
unstandardized,  group-oriented,  and  limited  to  small  numbers  of  students. 

Objective 

Military  training  managers  need  guidance  to  translate  residential  classroom  instruction  into 
transportable  instruction.  Not  only  do  they  need  detailed  guidelines  to  help  them  plan  and  carry 
out  the  many  steps  in  developing  transportable  courseware,  they  also  need  advice  from  teams  of 
developers  who  have  completed  the  process. 

Approach 

A  team  of  training  developers  at  the  Navy  Personnel  Research  and  Development  Center  put 
together  a  number  of  recommendations  ("lessons  learned")  based  on  their  experience  in 
converting  residential  courseware  to  transportable  courseware.  The  recommendations  address 
many  of  the  typical  problems  associated  with  the  instructional  development  process  and  suggest 
practical  solutions  to  them. 

Results 

Over  70  recommendations  are  presented  that  cover  all  phases  of  the  courseware  design  and 
development  process.  The  topics  range  from  team  assignments  to  lesson  specifications,  from 
authoring  aids  for  computer-based  instruction  to  evaluation  of  student  performance. 

Conclusions 

E)eveloping  transportable  courseware  is  a  complex  process.  However,  if  sufficient  time  is 
devoted  to  planning  and  preparation,  many  of  the  difficulties  typically  encountered  in  developing 
courseware  can  be  averted  or  minimized. 
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L  INTRODUCTION 


A.  THE  GTET  PROJECT 

This  report  documents  some  lessons  learned  during  the  prototype  lesson  design  and 
development  phase  of  the  GTET  project  GTET,  or  Guidelines  for  Transportable  Education  and 
Training,  is  funded  by  the  Joint  Services  Manpower  and  Training  Systems  Development 
Program.  One  major  objective  of  GTET  is  to  develop  guidelines  for  training  managers  who  are 
converting  residential  courses  into  more  cost-effecdve  transportable  ones. 

The  services  have  large  and  diverse  populations  of  personnel  who  work  at  many  locations 
and  require  training  in  a  variety  of  subject  areas.  Non-residential  instruction  can  provide  more 
standardized  instruction  than  conventional  instruction  if  the  content  is  systematically  designed 
and  presented.  Transportable  materials  can  reach  more  people  more  cost-effectively  than 
residential  instruction  because  they  can  be  duplicated  for  any  number  of  additional  students. 
These  conveniently  packaged  lessons  can  also  be  used  as  refresher  courses.  Transportable 
lessons  are  also  time-efficient  because  they  can  be  structured  to  adapt  to  individual 
skillA:nowledge  levels.  Moreover,  they  can  be  attended  to  when  it  is  most  convenient  for  the 
student 

Currently,  most  acquisition  management  courses  are  conducted  in  conventional  classrooms 
that  are  limited  in  size  and  in  their  availability.  Flexibility  of  the  instructors  to  adapt  to 
individual  students  is  likewise  limited.  Also,  classroom  instraction  varies  greatly  from  teacher  to 
teacher  and  even  from  semester  to  semester  for  the  same  teacher.  These  classroom  courses  also 
require  students  to  travel  to  and  live  in  locations  distant  from  their  work  sites,  sometimes  for  as 
long  as  20  weeks.  The  per  diem  costs  and  associated  labor  costs  due  to  time  away  from  work  are 
considerable. 

One  product  of  the  GTET  effort  is  the  document  Guidelines  for  Transportable  Education 
and  Training  (GTET)  (in  preparation),  which  is  intended  to  lead  managers  through  the  complex 
process  of  converting  residential  courseware  into  transportable  courseware.  These  Guidelines 
are  especially  useful  for  developing  nontechnical  courses,  such  as  those  concerned  with 
acquisition  management.  Acquisition  management  education  was,  in  fact,  selected  as  the 
content  area  to  be  explored  during  prototype  development  of  transportable  lessons.  This  area 
was  chosen  because  of  the  need,  as  noted  by  Congress,  the  Office  of  the  Secretary  of  E>efense, 
and  the  ncilitary  services,  to  improve  the  way  the  military  acquires  weapons  and  other  supplies. 

Participants  in  developing  and  evaluating  the  guidelines  and  training  materials  include  the 
Navy  Personnel  Research  and  Development  Center  (NAVPERSRANDCZEN),  the  Defense 
Systems  Management  College,  the  Army  Materiel  (Command,  the  Air  Force  Institute  of 
Technology,  the  Air  Force  Systems  Acquisition  School,  Naval  Air  Systems  Command,  Naval 
Sea  Systems  Command,  and  the  Army  Training  and  Doctrine  Osinmand. 
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B.  PLRPOSE  AND  ORGANIZATION  OF  THE  REPORT 


This  report  describes  the  lessons  learned  by  a  team  of  NAVPERSRANDCEN  instructional 
developers  while  planning  and  producing  transportable  instructional  materials  for  adult  learners. 
During  the  effort,  varied  procedures  were  used,  with  some  discarded  in  favor  of  others.  This 
repon  is  an  attempt  to  help  other  developers  of  transportable  (also  called  exportable,  distant, 
distributed,  portable,  correspondence)  educational  products  work  through  the  often  complex 
process  of  creating  quality  instructional  materials. 

The  repon’s  organization  generally  follows  that  of  the  Systems  Engineering  for  Instructional 
Development  (SEID)  process  (i.e.,  analysis,  design,  development,  implementation,  and 
evaluation),  which  is  a  variation  of  the  Instructional  Systems  Development  (ISD)  process.  Since 
the  SEID  and  other  versions  of  the  ISD  are  well  documented  elsewhere  (see  Appendix  A  for  an 
overview  of  the  SEID  approach  and  Appendix  B  for  a  list  of  selected  references  on  ISD-related 
publications),  we  make  no  attempt  here  to  present  a  comprehensive  overview  of  the  underlying 
process  used.  Also,  we  do  not  separately  address  the  implementation  phase,  which  is  a  part  of 
both  the  SEID  and  ISD  processes.  Since  we  were  not  the  implementers  in  this  instance  and  since 
lessons  learned  about  implementation  were  relatively  few,  we  included  what  we  had  in  the 
evaluation  phase  section. 

The  flow  charts  that  appear  in  the  repon  are  included  only  to  illustrate  the  topics  covered  in 
the  report.  They  do  not  represent  models  of  the  entire  instructional  development  process. 

Some  of  our  recommendations  mirror  those  found  in  ISD  sources.  We  repeat  them  here  to 
underline  their  importance.  Some  of  our  other  lessons  learned  deal  with  small  practical  details 
that  might  be  overlooked  in  more  theoretical  writings. 

We  begin  with  a  discussion  of  personnel  issues,  in  particular  the  setting  up  of  the  team  that 
will  create  and  develop  the  transportable  instruction.  There  are  some  issues,  such  as  planning 
and  communication,  that  apply  to  the  whole  process  and  not  just  to  one  category.  Lessons 
learned  about  these  issues  are  discussed  in  the  first  section  on  team  functioning  and  elsewhere 
where  they  are  relevant.  The  discussion  then  moves  to  the  general  topics  of  analysis,  design, 
development,  and  evaluation.  We  grouped  the  recommendations  within  these  broad  categories. 

Although  there  are  many  ways  to  present  courseware  (such  as  print  documents,  computer- 
based  instruction,  interactive  videodisks,  lectures,  simulators,  large  screen  projectors),  this  report 
focuses  on  the  media  used  in  our  effort-the  print  medium  and  the  computer-based  medium. 
However,  some  of  the  recommendations  can  apply  to  other  media. 
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II.  TEAM  FUNCTIONING 


Since  the  SE/ISD  process  usually  involves  a  team  effort,  one  interest  of  the  GTET  project  at 
NAVPERSRANDCEN  was  to  investigate  the  composition  of  development  teams.  The 
development  of  educational  materials  is  an  iterative  process,  requiring  various  skills.  There  are 
many  roles  a  member  can  fill  on  a  team,  and  a  single  individual  may  fill  one  or  more  roles, 
depending  on  need  and  capability. 

The  project  team  for  the  NAVPERSRANDCEN  effort  consisted  of  three  functional  groups-- 
instructional  developers  (both  experienced  and  novices),  subject  matter  experts  (SMEs),  and 
coordinators/facilitators  (Figure  1).  Our  recommendations  about  project  teams  are  based  on  our 
experience  with  these  groups. 


A.  INSTRUCTIONAL  DEVELOPERS 

Since  one  person  may  assume  one  (or  all)  of  the  traditional  SEED  or  ISD  roles,  hereinafter 
referred  to  collectively  as  "SE/ISD,"  the  term  "instructional  developer,  or  "ID,"  will  be  used  in 
this  repon  to  refer  to  all  the  traditional  SE/ISD  roles  of  analyst,  designer,  author,  and  evaluator. 
Essentially,  all  roles  other  than  the  SME  role  and  coordinator/facilitator  role  fall  under  the 
umbrella  of  ED.  A  description  of  the  traditional  ID  roles  within  the  context  of  the  SE/ISD 
process  is  presented  in  Appendix  A. 

At  NAVPERSRANDCEN,  a  core  group  of  two  to  four  developers  composed  of  an 
experienced  ID  and  novice  IDs  (student  contractors)  participated  at  all  levels  of  the  development 
effort,  from  planning  and  scheduling  to  proofreading  and  copying  of  computer  disks.  The 
student  contractors  had  some  experience  with  microcomputers,  but  none  developing 
transportable  materials.  They  were  trained  to  program  the  actual  content  of  the  lessons  into  two 
versions  (a  print  version  and  a  compurerizeo  version)  from  the  lesson  specifications.  A  second 
team  was  composed  of  outside  IDs  who  developed  several  of  the  lessons,  both  print  and 
computer  versions.  (The  decision  to  use  outside  contractors  to  fill  team  roles  is  normally  based 
on  the  availability  and  experience  of  personnel  as  well  as  on  budgetary  concerns.) 

Lessons  learned  are  presented  below  in  the  form  of  recommendations.  Those  directly  below 
focus  on  the  two  major  concerns  of  an  experienced  ID--assigning  project  tasks  and  training 
others  as  IDs  (see  Figure  2). 
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Figure  1 .  Formation  of  an  instructional  developmer  project  team. 


Figure  2.  Tasks  typ’cally  undertaken  by  instructional  developers. 
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1.  Assigning  Project  Tasks 


Recommendation:  IDs  should  deflne  the  tasks  of  lesson  specification  designers  and  lesson 
authors  before  they  uegin  lesson  construction. 

Task  assignments  for  the  lesson  specificution  designers  and  lesson  authors  largely  depend  on 
the  number,  skills,  and  experience  of  available  personnel.  The  ID  needs  to  decide  which  people 
will  be  assigned  to  which  tasks  before  determining  how  long  each  task  will  take  and  when 
milestones  can  be  reached. 

a.  Lesson  SpeciHcation  Assignments 

If  the  same  author  develops  both  the  lesson  specifications  (design  phase)  and  the  le..son 
content,  inconsistencies  between  them  will  be  easier  to  spot.  Also,  by  "'orking  closely  with  the 
topics  involved,  the  author  will  eventually  become  a  qjasi-expen  on  the  material. 

When  economic  or  personnel  constraints  dictate  that  different  individuals  develop  the  lesson 
specifications  and  corresponding  lesson  content,  then  the  lesson  content  author  must  take  the 
time  to  become  thoroughly  familiar  with  the  specifications.  In  our  effort,  outside  contractors 
prepared  the  lesson  specifications,  with  some  of  the  lessons  then  developed  by  in-house 
personnel  and  some  by  the  contractors.  It  took  some  time  for  the  in-house  team  members  to 
become  fajmiliar  with  the  specifications,  and  it  increased  the  need  for  the  SMEs  (professors  at  the 
Defense  Systems  Management  College)  to  provide  feedback  on  the  lesson  materials  to  ensure 
that  the  lessons  accurately  depicted  the  specifications. 

b.  Authoring  Assignments  Using  One  Medium 

If  only  a  print  medium  is  used  the  same  author  can  usually  complete  all  the  text  content  of  a 
single  lesson  if  the  lesson  is  not  excessively  long.  However,  if  the  one  medium  is  computer- 
based  or  uses  interactive  videodisk,  it  might  require  the  knowleoge  of  several  software  packages 
(for  ey  'ple,  a  word  processor  to  create  the  .exi,  an  authoring  system  to  translate  it  into  a  lesson, 
a  r"  ICS  package  '’sr  pictures).  In  this  case,  either  one  person  must  be  able  to  work  with 
se-  ural  software  packages  and  methods  or  several  people  must  work  on  different  aspects  of  each 
lesson. 

c.  A  uthoring  Assignments  Using  More  Than  One  Medium 

If  more  than  one  version  is  being  developed  (such  as  a  print  version  and  a  computer-based 
version),  several  approaches  are  possible,  and  each  has  certain  advantages. 

Same-Author  Method.  For  some  of  our  lessons,  the  same  author  completed  the  text  of 
both  the  print  and  computer  versions.  These  authors  developed  good  relationships  with  me 
relevant  SMEs  and  became  knowledgeable  about  the  content  and  hov'  best  to  present  it.  This 
method  also  works  well  because  an  author  is  more  likely  to  spot  inconsistencies  between  the 
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print  and  computer  versions  during  development.  Tlie  optimum  approach  would  be  to  have 
personnel  able  to  work  with  several  media  who  can  individually  work  through  both  versions. 

Different'Author  Method.  For  other  lessons,  different  authors  completed  the  texts  for  the 
print  and  computer  versions.  This  option  proved  most  efficient  when  time  was  limited  and  some 
IDs  were  new  to  the  software  packages.  If  training  a  team  member  on  how  to  use  the  software  is 
going  to  take  too  long,  it  might  be  more  efficient  to  let  the  trained  team  members  develop  the 
computerized  versions,  while  that  team  member  develops  the  print  versions. 

d.  Graphics  Assignments 

Recommendation:  At  least  one  team  member  should  be  skilled  at  creating  graphics. 

In  determining  this  assignment,  consider  the  artistic  ability  of  lesson  developers.  Be  aware 
that  some  people  do  not  have  sufficient  interest  or  ability  to  make  good  detailed  drawings,  even 
with  a  computer  program  as  an  aid.  Therefore,  in  choosing  team  members,  look  for  someone 
with  graphics  skills.  One  option  is  to  have  an  experienced  graphics  person  create  all  or  most  of 
the  graphics  for  the  lessons.  We  used  this  option  several  times. 

If  no  team  member  is  especially  skilled,  or  if  several  team  members  have  graphics  skills,  it 
might  be  more  practical  to  have  each  person  create  his^er  own  graphics.  When  using  this 
option,  detailed  guidelines  on  style  and  format  should  be  set  up  to  ensure  consistency  across 
lessons. 

e.  Computer  Assignments 

Recommendation:  At  least  one  team  member  should  be  adept  at  computer  use, 

specifically  in  terms  of  operating  systems  and  hardware. 

Computer  glitches  can  and  will  occur.  Hardware  problems,  such  as  faulty  disk  drives,  and 
software  tasks,  such  as  installing  and  learning  new  graphics  packages,  can  be  more  readily 
accomplished  by  a  team  member  familiar  with  the  more  technical  aspects  of  computers.  In  some 
organizations,  a  suppon  staff  provides  these  services.  Because  these  personnel  may  not  be 
available  at  critical  times,  we  recommend  that  one  member  of  the  team  be  able  to  create  and  run 
batch  files,  install  graphics  boards,  use  a  computer  operating  system,  troubleshoot  minor 
hardware  problems,  set  up  an  interface  between  a  computer  and  a  printer,  etc. 

f.  General  Assignments 

Recommendation:  Backup  personnel  should  be  available  for  assignments. 

It  is  worthwhile  to  identify  people  on  the  team  who  could  serve  as  backups  and/or  to  train 
new  members  in  a  varie  ■  of  skills.  For  example,  if  only  one  team  member  is  knowledgeable 


6 


about  the  authoring  system  or  the  graphics  package,  unavailability  of  this  member  for  any  reason 
could  delay  the  project.  We  therefore  recommend  cross-training  so  that  at  least  two  people  are 
knowledgeable  about  the  various  tasks,  including  technical  skills  required  for  using  various 
software  packages. 

2.  Training  Novice  IDs 

Recommendation:  The  team  leader  needs  to  determine  what  and  how  much  a  novice  needs 
to  know  to  perform  adequately  in  each  technical  area  and  how  long  it  will  take  to  train  a 
novice  to  perform  the  various  tasks. 

Unless  an  organization  has  a  team  of  IDs  experienced  in  all  aspects  of  lesson  construction 
(including  the  media),  some  training  will  be  necessary.  Training  should  cover  the  (a)  ISD 
process;  (b)  basics  of  computer-based  instruction  and  screen  design;  (c)  word  processing  system; 
(d)  lesson  authoring  system,  and  (e)  graphics  programs. 

The  learning  time  for  each  will  vary  according  to  the  complexity  of  the  programs  as  well  as 
the  experience  and  motivation  of  each  learner.  Even  with  software  programs  that  help  a  new 
user  learn  its  tools  quickly,  it  will  take  time  before  quality  products  can  be  created.  The  time 
required  to  train  novices  will  affect  the  product  delivery  schedule  as  well  as  the  quality  of  the 
final  product. 

Recommendation:  All  IDs,  whether  they  are  experienced,  novice,  or  contractor  personnel, 
must  be  provided  with  copies  of  applicable  standards  and  references  before  beginning 
lesson  construction. 

These  materials  should  be  supplied  to  all  team  members,  even  if  they  are  experienced  in 
lesson  construction,  and  to  contractors.  Periodic  reviews  of  their  work  should  be  scheduled. 

Recomnisndation:  When  employing  contractors,  an  organization  should  specify  in  the 
contract  that  contractor  personnel  must  have  certain  capabilities  and  experience  to  work 
on  the  effort. 

If  contractors  must  have  certain  skills  to  be  effective  team  participants,  these  skill 
requirements  should  be  included  in  the  training  plans. 


B.  SUBJECT  MATTER  EXPERTS 

The  subject  matter  expert  (SME)  serves  as  the  consultant  to  the  IDs  regarding  the  adequacy 
and  validity  of  the  lesson  content  and  test  items.  The  SME  provides  the  expertise  needed  to 
ensure  that  the  content  of  the  instruction  and  test  items  is  correct  and  presented  clearly, 
effectively,  and  with  suitable  detail  to  accomplish  the  instructional  goals.  In  our  effort,  SMEs 
(who  were  located  at  a  resident  instruction  school  at  a  remote  site)  provided  the  basic 
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instructional  materials  that  they  used  in  teaching  classroom  versions  of  the  lessons.  They  also 
reviewed  and  provided  feedback  on  the  developing  lesson  materials. 

Over  time,  the  SME  will  become  a  novice  ID,  and  the  ID  will  become  a  novice  SME.  When 
this  happens,  communication  becomes  more  effective  and  efficient.  How  quickly  this  occurs 
depends  on  many  things,  from  the  experience,  motivation,  and  background  of  the  individuals  to 
the  difficulty  or  complexity  of  the  subject  matter.  An  SME  needs  to  accept  the  fact  that  an  ID 
will  have  some  ideas  that  may  seem  foreign,  but  yet  may  work  very  well.  Likewise,  an  ID  needs 
to  be  prepared  for  the  fact  that  the  SME's  knowledge  and  terminology  may  be  difficult  to  grasp. 

The  SMEs  play  a  crucial  role  in  the  whole  process  of  developing  instructional  materials. 
The  more  open  the  communication  between  the  SMEs  and  the  IDs,  the  better  the  products.  We 
recommend  the  following  actions  to  ensure  that  communication  remains  open  throughout  the 
process. 

Recommendation:  The  team  leader  should  establish  from  the  outset  who  the  SMEs  will  be 
for  each  lesson. 

Each  lesson  should  have  at  least  one  SME  who  is  the  main  source  of  information  for 
establishing  and  reviewing  the  content  of  the  lesson. 

SMEs  also  need  to  understand  from  the  stan  their  role  on  the  development  team.  SME 
involvement  during  the  earliest  phases  should  help  ensure  the  adequacy  of  the  lesson 
specifications  and  thus  ease  the  process  of  developing  the  instructional  materials. 

Recommendation:  Before  beginning  lesson  development,  the  team  leader  should  determine 
how  available  each  SME  will  be  to  the  IDs. 

Each  SME's  calendar  needs  to  be  coordinated  with  the  lesson  development  calendar.  If 
sufficient  time  is  unavailable,  the  team  leader  should  either  negotiate  for  more  time  or  consider 
finding  a  different  SME.  And,  since  SMEs  are  usually  working  full  time  at  other  jobs,  it  is 
important  to  set  up  a  communication  system  that  is  nonintrusive  and  easy  to  use  so  that  SMEs 
and  IDs  can  exchange  information  quickly  during  lesson  production.  Ideally,  SMEs  work  at  the 
same  location  as  the  lesson  developers. 

Recommendation:  If  at  all  possible,  the  same  SMEs  should  be  used  throughout  course 
development. 

For  the  sake  of  continuity,  the  team  leader  should  try  to  employ  SMEs  who  arc  able  to 
support  the  project  for  its  duration.  A  change  of  SME  midway  through  design  or  development 
could  change  the  direction  of  a  lesson,  possibly  lengthening  the  entire  development  process  and 
forcing  the  DD  team  to  miss  due  dates.  For  one  of  the  lessons  developed  at 
NAVPERSRANDCEN,  SMEs  changed  twice,  once  during  the  lesson  development  and  again 
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during  the  review  process.  One  new  SME  saw  the  purpose  and  direction  of  the  lesson  as 
significantly  different  from  that  of  the  original  SME.  These  SMEs  had  different  instructional 
philosophies  and  differed  in  their  view  of  what  the  lessons  should  accomplish.  Unfortunately, 
the  change  happened  after  early  drafts  of  the  lesson  had  already  been  completed.  This  change 
complicated  the  lesson  development  and  extended  the  time  needed  to  complete  the  lesson. 

Recommendation:  The  team  leader  should  outline  what  each  SME  is  expected  to 
contribute  to  the  effort. 

The  SMEs  need  to  understand  from  the  beginning  what  types  of  materials  and  expertise  they 
are  being  asked  to  provide.  This  helps  define  the  role  of  the  SME  and  eliminates  confusion 
about  what  is  expected. 

Recommendation:  The  team  leader  should  designate  one  SME  who  has  the  authority  to 
give  final  approval  to  a  lesson. 

This  designation  is  especially  important  if  more  than  one  SME  is  involved  in  the 
development  of  a  lesson  or  if  lessons  draw  on  SMEs  from  several  disciplines. 

Procedures  should  be  established  as  early  as  possible  by  the  team  leader  to  help  SMEs 
reconcile  differences  over  lesson  content  or  specifications. 

To  illustrate:  One  set  of  our  lessons  covered  varying  topics  and  relied  on  SMEs  from 
different  educational  depanments.  There  was  no  coordinator  selected  to  oversee  these  various 
contributions.  This  module  did  not  have  the  consistency  of  thought  that  other  lessons  did  that 
either  originated  in  a  single  department  or  were  shaped  by  a  single  SME. 

Recommendation:  The  SMEs  should  be  encouraged  to  become  integrally  involved  in  the 
development  of  the  instructional  materials  by  helping  them  see  themselves  as  the  lesson 
content  developers. 

Describe  the  role  of  the  ID  to  the  SME  as  the  presenter  and  organizer  of  the  lessons,  that  is, 
the  ID  actuates  the  SME  s  methods  and  knowledge  according  to  instructional  development 
principles.  Each  party  should  view  the  process  as  a  joint  effort.  For  example,  the  term  "our 
materials"  should  be  used  instead  of  "your  materials"  or  "my  materials."  The  fact  that  the  whole 
team  owns  the  lessons  should  be  emphasized. 


C.  COORDINATORS/FACILITATORS 

Coordinator/facilitator  activities  focus  on  locating  and  coordinating  students/evaluators  who 
can  use  the  instructional  materials  systematically  and  provide  feedback  on  their  quality.  For 
both  the  formative  and  summative  evaluations,  coordinators  from  all  of  the  military  services 
participated  by  arranging  for  students/evaluators  and  instructors  to  go  through  the  lessons  and  fill 
out  feedback  forms  assessing  the  materials. 
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Many  of  the  recommendations  regarding  communication  between  IDs  and  SMEs  apply  here 
as  well.  In  fact,  SMEs  may  also  serve  the  role  of  coordinator/facilitator.  The  reader  is  also 
directed  to  the  evaluation  phase  section  of  this  report,  which  continues  the  discussion  of 
coordinators/facilitators. 


III.  ANALYSIS  PHASE 

Thorough  front-end  analysis  is  one  of  the  most  important  activities  in  the  process.  It 
describes  the  current  course  or  program  systematically  and  sets  the  foundation  for  the  lesson 
specifications.  Complete  and  accurate  lesson  specifications,  in  turn,  help  to  ensure  valid  lesson 
content. 

The  SE/ISD  approach  emphasizes  that  the  analysis  and  design  phases  must  be  completed 
before  courseware  is  developed  (see  Figure  3).  Within  the  analysis  phase,  the  current  course  is 
reviewed  and  a  determination  is  made  of  what  changes  are  needed  (see  Figure  4). 

Because  contractors  conducted  the  analysis  for  our  project,  we  do  not  have  many  first-hand 
lessons  learned;  however,  we  do  have  some  general  recommendations  to  make. 

Recommendation:  Sufficient  time  should  be  scheduled  for  the  front-end  analysis. 

This  is  vital,  since  a  thorough  analysis  is  a  time-consuming  part  of  the  SE/ISD  process. 

We  cannot  advise  the  reader  about  the  exact  amount  of  time  required  because  it  depends  on 
factors  such  as  the  state  of  the  existing  documentation,  the  complexity  of  the  tasks  and  concepts 
to  be  taught,  and  the  availability  of  SMEs. 

Recommendation:  Analysts  need  to  learn  about  the  lesson  users. 

Analysts  should  carefully  describe  the  learner  population  so  that  the  level  of  depth  and  detail 
of  course  content  can  be  set  appropriately.  The  most  useful  information  describes  the  learner's 
range  of  experience  and  education. 

Recommendation:  The  analyst  (who  could  be  the  team  leader  or  one  of  the  IDs)  should 
obtain  all  documents  and  content  resources  (e.g.,  reference  manuals,  standards  and 
specifications,  user  manuals)  that  will  be  needed  by  team  members. 

The  more  resource  materials,  references,  and  other  documentation  available  to  the  IDs,  the 
better.  The  SME  is  often  the  best  source  of  lesson  content  information.  All  team  members 
should  be  working  from  the  same  set  of  materials  (same  dates,  same  versions),  and  all  materials 
should  be  the  most  current  and  most  accurate  available. 


10 


Figure  3.  Flow  chart  illustrating  the  relationship  of  the  four  phases  of  the  SE/ISD  process. 


Figure  4.  General  steps  to  follow  in  the  analysis  phase. 
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IV.  DESIGN  PHASE 


In  general,  designers  use  the  results  of  the  analysis  phase  to  det^mine  the  media, 
instructional  strategies,  testing  strategies,  and  management  strategies  most  appropriate  to  the 
tasks  to  be  learned.  Our  lessons  learned  for  the  design  phase  cover  the  creation  of  lesson 
specifications  and  design  tools  to  aid  in  the  overall  design  and  development  process.  First, 
however,  we  make  one  general  process  management  recommendation. 

Recommendation:  Designers  should  plan  for  the  unexpected. 

When  planning  and  scheduling  the  entire  effort,  time  lines  should  be  generous  to  allow  for 
the  unexpected,  such  as  (a)  permanent  departure  of  employees;  (b)  hardware  failures;  (c)  funding 
cuts;  (d)  unanticipated  problems  with  SMEs;  (e)  unarticipated  problems  with  any  team  member 
or  with  lessons  materials,  etc. 


A.  CREATING  LESSON  SPECIFICATIONS 

The  outcome  of  the  design  phase  is  the  development  of  lesson  specifications.  Generally, 
lesson  specifications  contain  an  outline  of  the  lesson  structure,  the  lesson  objectives,  test  and 
practice  items,  examples,  the  essential  content  information  of  the  lesson,  and  instructional 
strategies.  Appendix  C  contains  sample  lesson  specifications  used  in  our  effort,  and  Appendix  B 
contains  ISD  references  that  outline  further  what  lesson  specifications  should  contain. 

First  we  will  discuss  some  overall  recommendations  that  deal  with  lesson  specification 
issues.  Then  we  will  present  lessons  learned  about  decisions  that  must  be  made  when  creating 
lesson  specifications— selecting  which  media  to  use,  selecting  hardware,  selecting  software, 
structuring  the  materials,  and  setting  up  the  testing  strategies  (Figure  5). 


INPUT  OUTPUT 


Figure  5.  Decisions  to  be  made  in  developing  lesson  specifications. 
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1.  Getting  Started 

Recommendation:  The  team  leader  should  communicate  to  team  members  and  SMEs  the 
purpose,  use,  form,  and  extent  of  lesson  specifications. 

If  any  team  members  are  unfamiliar  with  how  specifications  are  developed  and  what  they 
are  used  for,  they  might  not  adequately  review  the  specification  outline,  which,  in  turn,  will 
affect  their  products. 

Since  SMEs  contribute  significantly  to  the  process,  they  must  also  be  informed  about 
specifications.  When  the  SMEs  help  design  and  complete  the  specifications,  the  team  leader 
should  explain  that  the  specifications  must  be  extensive,  detailed,  and  valid,  and  that  the  content 
may  be  expanded  upon  or  deleted  as  need  be  as  lessons  progress  through  the  development  phase. 
Sketchy  specifications  or  ones  that  leave  issues  open  to  interpretation  will  cause  future 
difficulties  that  will  have  to  be  resolved.  Lesson  structure  should  be  clearly  delineated,  testing 
strategies  should  be  thoroughly  covered,  test  and  practice  items  should  be  developed  based  on 
the  lesson  objectives,  and  examples  should  be  clearly  and  fully  described. 

The  team  leader  should  stress  to  all  team  members  that  accuracy  and  completeness  of  lesson 
specifications  are  crucial  to  developing  lessons  that  are  accurate  and  complete.  Until  the  team 
agrees  that  the  specifications  are  complete  and  valid,  development  cannot  (or  should  not) 
continue. 

Appendix  C  contains  part  of  a  lesson  specification  we  used.  Note  its  limitations:  Although 
it  includes  samples  of  examples  and  test  items,  ideally  it  should  have  provided  specific,  detailed 
examples  as  well  as  the  necessary  test  items. 

Recommendation:  The  team  should  design  the  lesson  specifications  just  prior  to  the  lesson 
development. 

No  more  than  a  few  months  should  elapse  between  the  development  of  the  specifications 
and  the  development  of  the  lessons.  This  is  especially  important  for  lesson  content  that  might 
change  over  time. 

Specifications  become  outdated  for  many  reasons.  Curriculum  advisors  change  their  minds 
about  what  to  teach  or  how  to  teach  when  it  comes  to  putting  across  a  particular  concept,  and 
changes  also  take  place  in  the  world  that  may  change  the  content  of  some  lessons.  If  lesson 
specifications  appear  to  be  out  of  date,  the  team  leader  should  consult  with  an  SME  about 
obtaining  updated  information. 
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2.  Selecting  the  Overall  Media 

One  of  the  decisions  to  be  made  when  creating  lesson  specifications  concerns  which  media 
to  use  to  create  and  to  present  the  lessons.  During  this  process,  there  are  practical,  real-world 
compromises  that  usually  occur.  Often,  decisions  are  not  based  on  pedagogical  grounds  alone. 
Interests  of  potential  users  may  come  into  play.  Our  recommendations  focus  on  the  print  and 
computer-based  media  used  in  the  NAVPERSRANDCEN  effort. 

Recommendation:  IDs  must  recognize  that  not  all  course  subjects  are  amenable  to  all 
media,  such  as  the  computer-based  medium. 

The  NAVPERSRANDCEN  team  was  directed  to  create  both  print  and  computerized 
versions  of  all  lessons.  Cost  is  an  important  consideration  in  making  these  decisions.  The 
benefits  of  having  two  versions  of  the  same  lesson  must  be  weighed  against  the  cost  of  their 
creation. 

There  are  no  formulas  that  course  designers  can  apply  when  selecting  one  medium  over 
another,  although  some  have  suggested  that  this  might  be  possible.  There  has  even  been 
speculation  that  some  types  of  media  (such  as  interactive  videodisk)  might  be  able  to  stimulate 
certain  kinds  of  learning  better  than  more  static  media.  However,  recent  analysis  (Clark,  1985; 
Clark  &  Sugrue,  in  press)  has  shown  that  most  of  the  differences  in  student  performance  among 
the  various  media  are  likely  to  be  due  to  uncontrolled  variables  such  as  instructional  strategies 
and  the  motivational  effects  that  result  from  the  novelty  of  using  new  media.  Therefore  the 
primary  factors  to  be  considered  when  selecting  media  remain  (a)  the  kinds  of  tasks  and  concepts 
to  be  taught,  (b)  the  instructional  strategies  necessary  to  teach  them,  and  (c)  the  costs  of 
producing  and  using  the  resulting  courseware. 

Another  constraint  is  time  (see  the  following  recommendation).  Creating  computer  lessons 
that  are  mere  "page-turners"  (i.e.,  the  lessons  are  not  made  interactive  or  adaptive  to  the  learner’s 
progress)  is  expensive  and  wastes  both  computer  and  personnel  time.  Conversely,  certain  types 
of  lessons,  such  as  those  requiring  that  learners  branch  to  different  outcomes  depending  on  a 
previous  answer,  are  well  suited  for  automation.  Choosing  a  medium  is  a  diffrcult  decision,  but 
it  must  be  made,  and  made  early  in  the  design  process. 

For  a  further  discussion  on  how  to  select  media,  the  reader  is  directed  to  the  list  of  ISD- 
related  references  in  Appendix  B. 

Recommendation:  When  developing  computer-based  instruction,  ID  teams  need  to  allow 
adequate  time  for  preparation  of  sophisticated  screen  designs  and  ''interactivity,"  the 
process  by  which  the  student  and  the  software  interact. 

Computer-based  courseware  generally  does  not  progress  sequentially  (i.e.,  page  by  page  as 
print  documents  generally  do).  Exercising  the  potential  of  the  computer-based  method  usually 


involves  more  rime  than  does  the  creation  of  print  lessons.  The  extra  rime  is  spent  incorporating 
more  complex  presentation  strategies  such  as; 

a.  Multiple  graphics  (which  are  sometimes  in  overlays)  or  animations  that  enhance  learning 
and  retention. 

b.  Menus  that  branch  to  various  areas  within  the  program. 

c.  Interactive  practice  sessions. 

d.  Learner  control  of  sequence  and  choice  of  material. 

e.  Interactive  assessment  and  adaptive  feedback. 

Even  if  interactivity  and  screen  design  are  relatively  simple,  it  is  still  necessary  to  program 
the  branches  between  screens,  between  screens  and  menus,  and  even  to  toggle  screens  back  and 
fonh.  Since  the  amount  and  quality  of  learner  interactivity  is  one  estimate  of  the  quality  of  a 
computer-based  lesson,  time  to  develop  a  high  quality  lesson  must  be  provided. 

Note,  however,  that  developing  good  print  versions  of  instructional  materials  is  also 
challenging,  and  interactivity  can  be  incorporated  into  this  medium  as  well. 

3.  Selecting  Hardware 

If  a  computer  is  going  to  be  used  during  any  aspect  of  the  courseware  development,  such  as 
for  word  processing  or  as  the  actual  medium  selected  for  lesson  presentation,  hardware 
requirements  must  be  determined.  IDs  must  then  ensure  that  the  necessary  hardware  is 
obtainable.  Part  of  this  decision  involves  determining  what  off-the-shelf  hardware  is  available  or 
deciding  if  some  special  equipment  configuration  needs  to  be  set  up. 

We  decided  to  use  the  microcomputer  (the  Zenith  248,  an  MS-DOS-based  system)  because 
it  was  the  system  available  to  both  the  ultimate  lesson  managers  and  to  the  subject  population  in 
the  field.  This  microcomputer  system  was  used  to:  (a)  produce  the  print  lessons,  (b)  interact  with 
the  computerized  authoring  system,  and  (c)  present  the  computerized  version  of  the  lessons. 

Recommendation:  Compuiei  space  requirements  (i.e.,  disk  storage  space  as  well  as 
memory  requirements)  must  be  addressed. 

These  requirements  will  depend  on  (a)  the  size  of  the  lessons  being  developed,  (b)  the 
amount  of  space  and  memory  needed  to  run  any  required  software  programs,  and  (c)  the  uses  to 
which  the  computer  will  be  put-for  development  and/or  presentation. 
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To  address  the  above  issues,  the  ED  team  should  use  the  results  of  the  analysis  phase  in 
determining  how  large  the  lessons  will  be.  Then,  the  team  should  investigate  the  storage  and 
memory  requirements  of  the  software  programs  to  be  used.  Space  and  storage  requirements  will 
be  affected  if  several  lessons  are  to  be  developed  on  one  computer.  Finally,  if  computer-based 
lessons  are  going  to  be  developed,  space  and  memory  requirements  of  the  authoring  system  need 
to  be  considered.  Space/storage/memory  considerations  may  be  less  of  an  issue  if  the  computer 
is  used  only  for  word  processing,  although  lesson  size  and  the  space/memory  requirements  of  the 
word  processing  system  must  be  considered. 

Thus,  if  lessons  are  large,  if  several  are  developed  on  one  computer,  and  if  the  lessons  are  to 
be  computer-based,  then  an  additional  hard  disk  drive  may  be  required  as  well  as  a  high  density 
floppy  disk  drive,  and/or  an  enhanced  memory  board. 

Recommendation:  If  graphics  are  necessary,  the  ID  team  should  determine  as  soon  as 
possible  what  kind  of  graphics  will  be  required  as  well  as  what  kind  of  graphics  board  and 
monitor  to  use  to  ensure  compatibility  with  its  own  computers  as  well  as  with  those  of 
potential  users. 

The  type  of  graphic  equipment  ranges  from  ega  (color  graphics  adaptor),  to  ega  (enhanced 
graphics  adaptor),  to  vga  (video  graphics  array),  and  other  capabilities  are  being  developed  all 
the  time.  Because  each  type  may  require  a  different  type  of  graphics  board  and/or  monitor  and 
since  some  systems  won't  run  lessons  created  on  a  different  system,  the  ID  team  should 
determine  early  if  potential  learners  have  the  needed  equipment  to  view  the  lessons. 

Recommendation:  If  floppy  disks  are  to  be  used,  the  ID  team  should  obtain  disks  that  are 
error-free. 

This  might  seem  a  small  and  obvious  point,  but  such  details  can  cause  serious  problems  and 
delay.  For  example,  faulty  floppy  disks  resulted  in  inoperable  lessons  at  one  point  during  our 
project.  To  avoid  this  problem,  the  team  should  test  different  brands  of  disks  to  determine  the 
most  reliable. 

4.  Selecting  Software 

In  our  project,  we  were  instructed  to  use  a  non-proprietary,  government-owned  authoring 
system.  This  selection  saved  money  since  the  government  didn't  have  to  buy  any  new  software 
or  pay  fees  for  distribution  to  training  sites.  However,  like  all  software,  the  package  had 
limitations  and  several  "bugs"  in  the  system  had  to  be  dealt  with  during  the  project.  When 
selecting  any  software,  the  ID  team  should  try  to  be  realistic  about  its  limitations  as  well  as  its 
useful  features.  This  approach  helps  prepare  the  team  to  be  flexible  when  developing  the 
courseware. 
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When  planning  to  use  computers  to  develop  courseware,  whether  print  documents  or 
computer-based  lessons,  three  or  more  types  of  software  systems  will  be  required:  (a)  a  word 
processing/desktop  publishing  system,  (b)  an  authoring  system,  and  (c)  a  graphics  package. 

Recommendation:  The  ID  team  members  need  to  ensure  that  the  software  systems  are 
compatible  with  one  another. 

For  example,  an  authoring  system  needs  to  be  compatible  with  the  hardware,  with  the 
graphics  package  (if  any),  and  with  the  word  processing/desktop  publishing  system,  if  the 
authoring  system  needs  to  interact  with  these  other  packages. 

a.  Interaction  Between  Authoring  System  and  Word  Processing  System 

If  the  authoring  system  must  interact  with  a  word  processing  system,  the  systems  must  be 
compatible.  Our  authoring  system,  for  example,  interacted  with  a  wide  variety  of  word 
processors.  Other  authoring  systems  are  very  specific  about  which  word  processed  text  they  will 
accept.  In  addition,  since  we  created  both  print  and  computer  versions  of  lessons,  and  since  we 
created  the  print  versions  first,  development  was  streamlined  because  our  word  processing 
program  could  readily  convert  text  to  the  authoring  program. 

b.  Interaction  Between  Graphics  Programs  and  AuthoringAVord  Processing  Systems 

The  graphics  editor  used  in  our  effort  (Dr.  Halo)  was  supponed  by  both  our  word  processing 
system  (WordPerfect)  and  our  authoring  system  (Computer-based  instruction  Authoring  Tools 
System  [CATS]).  However,  color  graphics  for  computer-based  lessons  sometimes  had  to  be 
modified  for  the  print  lessons  because  some  color  combinations  did  not  translate  well  to  black 
and  white.  Fonunately,  the  modifications  were  easily  accomplished  due  to  the  color  swapping 
capabilities  of  our  graphics  editor.  If  an  effort  involves  using  similar  graphics  for  two  media,  the 
graphics  must  be  able  to  support  both  media. 

c.  Interaction  Between  Software  and  Hardware  Systems 

It  may  seem  an  obvious  point,  but  not  all  software  is  compatible  with  all  hardware.  For 
example,  we  discovered  that  the  fonts  available  through  the  word  processing  system  we  used 
(WordPerfect)  were  not  supported  by  all  of  the  different  printers  we  had  available. 

Although  there  are  some  ways  to  circumvent  the  incompatibility  between  software  and 
hardware,  modifications  may  be  cumbersome  and  cause  delays  in  meeting  deadlines.  If 
budgetary  or  other  contract  restraints  require  the  use  of  a  software  package  that  entails  modifying 
the  printer  interface,  extra  time  for  this  correction  must  be  scheduled. 


5.  Establishing  the  Instructional  Strategy 

Recommendation:  The  ID  team  should  set  up  a  general  framework  for  the  lessons. 

We  recommend  that  ID  teams  organize  the  lesson  information  into  categories  or  sections 
(e.g..  Objectives,  Examples,  Practice  Items)  that  will  provide  structure.  This  is  particularly 
important  when  the  course  materials  will  be  used  by  people  who  have  a  bioad  range  of 
experience,  education,  and  interests,  since  students  can  use  this  organization  to  individualize 
their  learning  experience.  For  example,  students  who  want  only  a  refresher  couise  or  review  of 
the  topic  might  be  instructed  to  concentrate  on  the  Overview  and  Summary  sections.  Students 
who  require  more  thorough  training  would  study  additional  sections  that  present  information  in 
more  detail.  The  less  experienced  learner  should  be  provided  with  enough  resources  to 
adequately  accomplish  the  lesson  objectives. 

Organizing  information  for  all  lessons  by  category  or  section  lIso  makes  self-pacing  easier 
because  students  know  where  they  are  and  what  kind  of  information  they  are  receiving  at  any 
time.  They  can  rapidly  decide  what  to  do  next.  Such  organization  of  instructional  material  is 
learner-centered  because  it  allows  the  learner  to  choose  the  pace  and  sequence  of  oiesentation. 

For  example,  our  lesson  structure  included  the  following  categories; 

1.  Introduction-A  stateme-'t  of  the  lesson  objectives 

2.  Core  Information-The  essential  content  of  the  lesson 

X  More  Information-Further  details  and  descriptions  of  the 
core  information 

4.  Examples-Hypothetical  or  "real  life"  illustrations  or 
applications  of  the  core  information 

5.  Summary-A  brief  restatement  of  the  core  information 

6.  Practice  and  Feedback-Questions  and  feedback  about  the 
core  information 

Lesson  structure  can  change,  however,  as  development  and  evaluation  continue.  For 
example,  our  More  Information  category  proved  somewhat  ambiguous.  Questions  arose  as  to 
whether  this  category  contained  only  supplemental  information  and,  thus,  was  not  testable  in  the 
quizzes.  As  a  result,  we  used  the  More  Information  category  less  often.  We  moved  clarifying 
information  into  the  Core  Information  or  Examples  sections. 
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The  Table  of  Contents  in  the  printed  version  or  the  menu  in  the  computerized  version  of  a 
lesson  should  describe  categories  adequately  so  the  learner  can  quickly  decide  which  categories 
of  the  instruction  to  study  to  accomplish  the  lesson  objectives. 

Recommendation:  The  learner  is  the  customer. 

In  a  customer-oriented  organization,  everything  possible  is  done  to  accommodate  the 
customer,  within  this  context,  the  customer  is  the  student.  The  format  of  the  lessons  should  be 
user-friendly  and  structured  so  that  learners  always  knows  where  they  are  v  'thin  a  lesson  and 
what  to  do  with  any  component  of  it.  Directions  should  be  simple.  For  example,  at  the  end  of 
each  printed  page,  or  on  each  computer  screen,  it  should  be  clear  to  the  learner  what  to  do  next. 
Developers  should  keep  in  mind  that  the  learner's  task  is  to  leam  lesson  content,  not  to  struggle 
with  a  complicated  lesson  format. 

Recommendation:  The  learner  needs  feedback. 

Feedback  can  be  provided  to  users  through  various  assessments  (e.g.,  pre-  and  post-quizzes) 
and  through  exercises  and  examples.  These  mechanisms  also  promote  interaction.  However, 
quizzes  should  be  introduced  to  users  as  diagnostic  or  prescriptive  tools  and  not  just  assessments 
for  a  "score."  Questions  are  diagnostic  because  they  are  designed  lO  help  learners  determine 
strengths  and  weaknesses.  They  are  prescriptive  because  they  are  intended  to  guide  learners  to 
the  specific  materials  they  need  to  use  to  accomplish  the  lesson  objectives. 

Questions,  exercises,  and  examples  should  be  designed  so  that  the  learner  can  apply 
cognitive  skills  and  knowledge  appropriate  to  the  objectives.  They  should  not  be  simplistic 
requirements  that  ask  the  student  to  repeat  facts  or  bits  of  information  based  on  rote  memory, 
unless,  of  course,  the  lesson  is  intended  as  a  drill  to  facilitate  rote  memorization. 

Recommendation:  Appendices  for  print  lessons  and  users'  guides  for  computerized  lessons 
are  useful  additions  to  lessons. 

For  print  versions,  appendices  could  contain  the  most  important  content  points,  examples  of 
documents,  formulas,  charts,  and  diagrams  that  learners  could  use  later  on  thf  job  or  as  a  study 
aid  for  final  review  before  the  lesson  test. 

For  computer-based  lessons,  printed  users'  guides  that  accompany  each  lesson  could  contain 
the  above  job/study  aids  as  well  as  any  other  material  not  amenable  to  a  computerized  format. 
For  example,  some  sections  in  our  users'  guides  for  our  computer-based  lessons  included 
examples  that  were  extensive  and  more  easily  read  in  a  print  form  than  on  a  computer  screen. 

Users'  guides  and  print  version  appendices  could  also  contain  pages  that  list  the  learning 
objectives  for  each  lesson  with  the  suggestion  that  students  might  want  to  take  a  few  notes  as 
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they  move  through  the  lesson.  The  notes  would  serve  as  a  summary  of  the  lesson  content  and 
would  provide  a  study  guide  for  the  tests.  For  learners  using  a  computer-based  lesson,  it  would 
also  dispel  any  thoughts  that  writing  on  paper  is  somehow  "primitive"  and  to  be  avoided. 

Recommendation:  Instructions  that  are  standard  across  lessons  should  not  be  presented 
repetitively  to  students. 

The  user’s  guide  we  included  with  each  of  the  computerized  lessons  contained  information 
on  how  to  install  and  use  the  lesson.  The  installation  and  use  instructions  were  included  with 
each  lesson,  though  these  instructions  were  identical  for  every  lesson.  Our  purpose  was  to  make 
sure  the  lessons  could  be  used  independently.  However,  in  most  cases,  students  were  working 
through  a  whole  series  of  lessons  and  saw  some  of  the  same  instructions  in  each  user's  guide. 

Based  on  our  experience,  we  recommend  that  repetitive  information  such  as  instructions  be 
kept  in  a  separate  booklet  that  can  be  given  to  each  student  once.  Information  specific  to  each 
lesson  could  then  be  distributed  in  a  different  document  as  needed.  Students  would  not  receive 
redundant  material,  and  printing  and  mailing  costs  would  be  reduced. 

Recommendation:  For  the  computer-based  medium,  broad  guidelines  about  the 

complexity  of  the  computer  menu  need  to  be  set. 

Guidelines  should  be  flexible  enough  to  allow  the  lesson  author  some  creativity,  but 
structured  enough  to  ensure  that  students  know  where  they  are  within  the  lesson  and  are  able  to 
move  through  it  without  needing  to  leam  a  complicated  series  of  movements.  Menus  need  to 
serve  a  specific  purpose,  they  need  to  be  accessible  at  almost  any  point,  and  they  should  be 
invisible  unless  needed. 

For  example,  during  the  formative  evaluation  of  some  of  our  earliest  lessons,  we  learned 
that  some  of  our  nested  menus  were  confusing  to  learners.  (Nesting  menus  are  a  series  of  menus, 
with  one  leading  to  another  and  possibly  then  to  a  third  before  actually  presenting  lesson 
content.)  These  proved  be  a  hindrance  rather  than  a  help  to  learners'  progress  through  a 
lesson.  These  menus  were  deleted  from  our  computer-based  lesson  structure. 

6.  Setting  Up  Testing  Strategies 

Recommendation:  Test  items  and  practice  items  should  be  written  while  creating  the 
lesson  specifications. 

Test  items  should  follow  directly  from  the  lesson  objectives  and  test  only  the  knowledge, 
skills,  and  abilities  required  by  these  objectives.  When  test  items  and  objectives  are  developed  at 
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the  same  time,  the  process  is  simplified  and  far  more  efficient  For  example,  the  lesson 
specifications  developed  for  our  lessons  did  not  contain  the  complete  list  of  test  items;  it  proved 
to  be  difficult  2  years  later  to  create  effective  items  to  test  the  objectives. 

Recommendation:  A  variety  of  question  types  should  be  used  when  creating  both  practice 
items  and  quizzes/tests. 

Sometimes  external  constraints  can  affect  the  use  of  a  variety  of  question  types.  One  such 
constraint  is  project  funding-it  can  take  more  time  and  thus  more  money  to  develop  and  plan  to 
score  answers  to  open-ended  questions.  Another  constraint  is  that  the  computer  authoring 
system  may  be  more  difficult  or  time-consuming  to  program  (which  also  translates  into  funding) 
for  open-ended  question  types.  Our  lessons  generally  used  multiple  choice  items,  partially  due 
to  external  constraints  made  at  the  project  outset. 

We  also  considered  the  ease  of  item  preparation  and  ease  of  maintaining  and  scoring  items 
when  we  designed  questions.  However,  multiple  choice  items  alone  are  not  always  sufficient, 
even  though  they  are  the  most  easily  scored  by  computer  or  by  hand.  The  student  should  be 
presented  with  several  ways  that  test  their  understanding  of  the  content,  especially  in  practice 
items  included  within  the  lessons.  Matching  items,  shon-answer  items,  open-ended  questions  or 
questions  that  ask  the  student  to  solve  problems  similar  to  those  presented-all  of  these  contribute 
to  the  creation  of  quality  instructional  materials.  All  forms  of  questions  should,  of  course, 
directly  address  the  learning  objectives  stated  in  the  lesson  specifications. 


B.  CREATING  DESIGN  MANAGEMENT  TOOLS 

Design  management  tools,  such  as  flow  charts,  are  also  created  during  the  design  phase 
along  with  lesson  specifications.  Design  management  tools  can  help  ease  the  whole  process  of 
designing  and  developing  instructional  materials.  We  recommend  creating  flow  charts  and 
developing  file  management  templates  and  lesson  templates  as  tools  that  can  help  organize  the 
instructional  development  process. 

1.  Making  Flow  Charts 

Recommendation;  Flow  charts  are  useful  in  outlining  and  documenting  the  plan  and 
progress  of  the  development  effort. 

Flow  chans  are  management  tools  that  can  be  used  in  any  of  the  phases  of  instructional 
development  and  are  useful  at  most  stages  of  planning  and  managing  a  project.  We  found  them 
of  special  use  in  the  design  and  development  phases.  Flow  chans  can  be  used  to: 

a.  Plan  the  entire  lesson  construction  effort,  from  analysis  through  evaluation. 

b.  Pinpoint  areas  in  the  process  that  are  critical  to  timely  and  efficient  completion  of  the 
effort. 
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c.  Document  and  compare  the  planned  process  with  the  actual  pnx^ess  used. 

d.  Document  and  compare  the  actual  process  used  with  the  process  recommended  for  future 
development  effons. 

2.  Developing  File  Management  Templates 

We  developed  a  variety  of  ways  to  organize  and  track  computer  files.  These  files  can  be 
word  processing  files  (for  print  lessons)  or  authoring  system  files  (for  computer-based  lessons). 

Recommendations  on  file  management  will  cover  the  following  issues:  directory  structure,  file 
structure,  naming  conventions,  file  creation  dates,  and  file/disk  backups. 

a.  Directory  Structure 

Recommendation:  A  standard  by  which  to  structure  the  on-line  directories  should  be 
established  so  that  the  directories  reflect  the  hierarchy  or  outline  of  the  lessons  to  be 
developed. 

The  computer  is  a  powerful  tool  for  organizing  courseware.  A  standard,  well-organized 
directory  structure  will  help  establish  consistency  and  help  lesson  developers  keep  track  of  the 
development  process.  A  sample  of  a  directory  structure  that  reflects  the  outline  of  lessons  is 
presented  in  Appendix  D. 

b.  File  Structure 

Recommendation:  A  standard  for  on-line  file  structure  that  reflects  the  organization 
of  lessons  should  be  established. 

File  structure  will  vary  according  to  the  media  used  and  the  structure  of  the  lessons.  Since 
we  worked  only  on  print  and  computer  versions  of  lessons,  the  following  recommendations 
address  only  those  two  media. 

Recommendation  for  print  versions:  The  word  processing  file  for  a  print  version  of  a 
lesson  can  be  contained  in  either  a  single  file  or  a  number  of  smaller  files. 

This  decision  will  depend  on  lesson  organization,  preference  for  working  on  smaller  files 
versus  one  large  file,  whether  the  files  will  fit  on  one  or  more  floppy  disks,  and  on  the 
characteristics  of  the  word  processing  system.  For  example,  our  word  processing  system  would 
not  print  a  file  if  the  file  size  exceeded  half  the  floppy  disk  space.  If  files  are  split,  page  numbers 
need  to  be  tracked  to  avoid  gaps  in  numbering  at  final  printing. 


Recommendation  for  computer-based  versions:  For  the  computer-based  lessons,  we 
found  it  best  to  set  up  separate  files  to  correspond  to  functional  categories  of  the  lesson, 
such  as  Core  Information  and  Examples. 

These  files  were  shorter  and  easier  to  work  with  for  testing  with  our  authoring  system. 
Since  authoring  systems  vary  in  their  file  management  capabilities,  the  file  structures  might 
differ  to  accommodate  each  system. 

c.  Naming  Conventions  for  Directories  and  Files 

Recommendation:  The  naming  code  should  be  simple  and  logical. 

That  is,  the  directory  and  file  names  should  correspond  in  some  way  to  what  they  contain 
and  to  where  they  belong  in  the  sequence  of  the  courseware.  Examples  of  directory  and  file 
names  for  print  files,  computer  files,  and  graphics  files  used  in  our  project  are  presented  in 
Appendix  E. 

Recommendation:  The  numbering  system  should  provide  clues  to  the  learner. 

If  the  ID  team  members  are  using  several  levels  of  instruction,  such  as  major  modules  with 
smaller  lessons  within  those  modules,  they  should  use  a  different  numbering  scheme  for  each  to 
help  students  remember  the  level. 

For  example,  Module  I  could  contain  Lessons  A,  B,  and  C,  and  the  lessons  could  contain 
Chapters  1,  2,  and  3.  This  scheme  provides  an  extra  clue  to  the  learner  by  which  to  keep  track  of 
where  she  or  he  is  in  the  lesson  (i.e.,  letters  are  always  lessons;  chapters  are  always  arable 
numbers;  modules  are  always  roman  numerals). 

d.  File  Creation  Dates 

Recommendation:  Dates  on  files  should  be  kept  current  and  correct. 

Although  this  may  seem  trivial  and  obvious,  keeping  the  correct  dates  and  times  on  files  is 
important  for  identifying  the  most  current  files,  especially  if  several  team  members  process  some 
of  the  same  files. 

If  a  computer  is  used  for  word  processing  or  computer-based  authoring,  the  correct  date  and 
time  are  probably  recorded  automatically  via  a  working  time  clock  in  the  computer.  But,  if  the 
computer  does  not  automatically  keep  a  time  clock,  there  is  usually  a  manual  way  to  set  the 
clock  after  the  computer  boots  up.  The  operating  system  manual  should  describe  this  procedure. 

If  a  computer  is  not  used  to  process  the  documents,  the  creation  date  can  be  kept  track  of  by 
simply  writing  the  date  and  time  on  the  document. 
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e.  File/Disk  Backups 

Recommendation:  Backup  files/disks  should  be  created  regularly. 

This  is  an  elementary  point  but  it  cannot  be  overstressed.  Mistakes  and  accidents  can 
happen  io  computers  and  evcw  to  disks.  We  recommend  that  both  backup  disks  and  hard  copies 
of  documents  be  created  to  ensure  survival  of  the  product. 

The  frequency  of  creating  backup  material  depends  on  how  much  work  has  been 
accomplished  in  a  given  period  of  time.  If  team  members  are  woridng  steadily  over  a  week’s 
time  to  develop  a  lesson,  it  would  take  at  least  that  much  time  again  to  reproduce  those  efforts  if 
disks  turned  out  to  be  defective  or  inadvertently  destroyed.  Certainly  backup  material  should  be 
created  on  almost  a  daily  basis  when  production  of  the  ID  team  is  high. 

3.  Setting  Up  Lesson  Templates 

Recommendation:  Templates  are  a  useful  tool  for  lesson  development. 

One  of  the  major  decisions  to  make  before  writing  lessons  concerns  formatting.  This  may 
seem  at  first  to  be  working  backwards--after  all,  if  there  is  nothing  to  format,  what  difference 
does  it  make  what  typeface  will  be  used?  But  consider  what  happens  when  seven  lessons  are 
written  by  seven  authors,  each  with  a  unique  style  of  headers,  highlighting  conventions,  and 
abbreviation  conventions.  To  make  the  lessons  compatible,  at  least  six  of  them  will  probably 
have  to  be  meticulously  revised. 

To  solve  this  problem,  we  recommend  the  use  of  lesson  templates.  By  lesson  templates,  we 
mean  files  (either  for  print  or  computer  versions)  that  serve  as  prototypes  for  the  initial  standards 
established  for  lesson  development.  Templates  not  only  provide  the  entire  lesson  development 
process  with  structure,  but  help  improve  consistency  within  and  between  lessons  and  between 
different  media  versions  of  lessons.  If  at  all  possible,  complete  and  final  templates  should  be 
developed  first;  they  should  include  as  much  detail  as  possible  for  both  print  and  computer 
lessons. 

The  use  of  templates  does  not  have  to  be  unvarying.  A  different  format  might  be  used  for 
some  of  the  lessons  to  accommodate  special  instructional  purposes  or  to  arouse  student  interest 
with  occasional  visual  surprises. 

a.  Templates  for  Print  Versions 

Templates  for  print  versions  can  include  details  such  as  format  instructions  (margins, 
spacing,  tabs,  etc.),  highlighting  conventions,  typefaces,  wording  and  abbreviation  conventions, 
headers/footers,  and  color  restrictions,  if  they  apply.  These  templates  can  exist  in  a 
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computerized  file  (in  the  relevant  word  processing  format)  that  can  be  given  to  the  lesson  authors 
before  they  begin  to  write  the  lessons.  Appendix  F  presents  a  sample  of  a  format  document  for 
development  of  print  lessons,  one  we  used  in  our  own  effort. 

b.  Templates  for  Computer-based  Versions 

Templates  for  computer-based  lessons  should  include  standards  for  (1)  naming  storyboards 
(which  contain  the  text  and  other  programming  information  for  each  screen),  (2)  positioning 
menus  and  directional  cues  on  the  screen.  (3)  margins,  (4)  centering  cues,  (S)  color  standards  or 
restrictions,  (6)  branching  conventions,  and  any  other  screen  design  conventions.  Creating 
templates  for  computer-based  lesson  development  expedites  production  since  new  information 
can  be  "read  into"  a  set  of  predesigned  templates  and  changed  accordingly.  Appendix  G 
contains  printouts  of  a  template  file  used  for  the  computer-based  lessons  in  our  effort. 

c.  Printed  Reference  Copies  of  Templates 

A  printed  copy  of  both  the  print  version  templates  and  the  computer  version  templates 
should  also  be  available  to  lesson  authors  as  reference  documents.  For  example,  our  lesson 
authors  had  a  binder  of  the  printed  computer-based  templates  to  refer  to  when  standards  were  in 
question. 

d.  Changes  to  Templates 

Changes  to  templates  should  only  be  done  for  a  good  reason.  If  a  template  is  changed,  all  of 
the  materials  already  produced  will  have  to  be  revised  to  accommodate  tliis  change.  One  major 
change  we  made  to  our  computer-version  template  after  some  lessons  had  been  written  was  the 
addition  of  screen  numbers  to  identify  each  computer  screen.  We  decided  this  was  vital  for 
tracking  revisions  and  for  locating  where  we  were.  It  took  time  to  add  the  screen  numbers,  but 
we  decided  it  was  wonh  the  effort. 

4.  Maintaining  and  Updating  Lessons 

Recommendation:  The  ID  team  should  provide  methods  for  future  updates  and 
maintenance  of  the  courseware. 

Eventually,  the  lessons  will  need  to  be  changed.  Information  will  need  updating; 
instructors  will  want  to  reorganize  the  content,  shift  the  emphasis  of  a  lesson,  or  add  other  topics. 
Usually,  the  people  who  develop  the  lessons  won't  be  available  to  make  these  changes.  So,  part 
of  the  courseware  development  effort  must  include  creating  directions  for  future  lesson 
maintainers  so  that  they  can  make  changes  efficiently. 
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Some  courseware  provides  built-in  features  for  making  changes  and  additions. 
Directions  for  lesson  maintenance  are  often  included  in  a  teacher's  manual.  Other  lessons,  such 
as  those  created  in  the  NAVPERSRANDCEN  effort,  provide  separate  software  and  instruction 
manuals,  necessary  because  the  authoring  system  manual  did  not  include  sufficient  help  for 
editing  lessons.  In  either  case,  it  is  important  to  plan  ahead  for  future  maintainers  so  that  lessons 
can  be  changed  as  needed. 


V.  DEVELOPMENT  PHASE 

Development  involves  the  authoring  and  production  of  the  actual  instructional  materials.  If 
a  thorough  analysis  has  been  done  and  complete  lesson  specifications  have  been  produced  during 
the  design  phase,  development  should  proceed  smoothly.  Lessons  learned  for  the  development 
phase  address  preparing  to  develop  lessons,  developing  lesson  content,  reviewing  and  revising, 
and  producing  the  final  materials. 


A.  WORKING  EFFECTIVELY  WITH  THE  SMEs 

Recommendation:  If  the  lesson  specifications  are  not  detailed  enough,  the  SMEs  should  be 
asked  in  writing  for  additional  examples  and  instructional  strategies. 

Sometimes  lesson  specifications  are  not  sufficiently  detailed.  The  SMEs  are  the  best 
sources  for  additional  examples,  practice  items,  and  descriptions  of  teaching  approaches  for 
some  lesson  topics.  SMEs  should  provide  much  of  the  actual  wording  of  the  lessons  and  from 
their  experience  can  provide  many  suggestions  on  which  topics  need  more  attention  or 
explanation  to  make  them  clear  to  students. 

Recommendation:  A  signofT  procedure  should  be  established  whereby  the  SME  or  SME 
coordinator  formally  signs  and  dates  approval  of  each  lesson. 

As  the  ID  team  and  SMEs  review  the  lesson  materials,  the  team  should  establish  a  procedure 
for  documenting  the  SME  s  approval  of  each  lesson.  This  approval  process  will  help  instill  a 
sense  of  involvement  and  ownership  as  well  as  accountability  by  the  SME  or  SME  coordinator. 
The  signer  should  be  the  SME  with  final  approval  authority.  Signoffs  on  drafts  will  also  serve  as 
documentation  of  progress. 

Recommendation:  A  detailed  log  of  contacts  between  the  SMEs  and  the  IDs  should  be 
kept. 

Both  the  SMEs  and  IDs  should  keep  a  log  of  each  contact  with  one  another  throughout  the 
entire  SE/ISD  process,  but  it  is  particularly  important  during  the  development  phase.  This  log 
should  contain  brief  summaries  of  the  date,  duration,  and  content  of  the  contact,  and  whether  any 
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action  was  necessary  as  a  result  of  it.  These  data  are  valuable  because  they  document  what  has 
been  done  and  provide  guidance  for  future  effons. 


B.  DEVELOPING  LESSON  CONTENT 

Recommendation:  Lesson  authors  should  become  thoroughly  familiar  with  the  subject 
matter  before  beginning  to  write  and  must  closely  review  the  lesson  specificatims  for 
content  and  method  of  presentation. 

Lesson  authors  should  also  review  any  other  source  materials  such  as  readings  or  videotapes 
of  any  current  classroom  instructions.  If  the  content  is  particularly  complex  and  if  the  ID  team 
has  enough  time  and  resources,  lesson  authors  should  attend  classroom  lessons  on  the  subject 
matter  or  be  taught. 

Recommendation:  Descriptive  examples  and  practice  items  should  be  used  whenever 
possible  throughout  lessons. 

Some  of  the  students  who  evaluated  our  lessons  commented  that  they  preferred  having  a 
wide  variety  of  examples,  including  what  we  called  Practice  and  Feedback  items.  If  time  had 
allowed,  we  could  have  included  more  examples.  It  would  not  have  had  a  negative  impact  on 
the  time  required  for  learners  to  finish  the  lesson,  since  lessons  were  structured  so  that  learners 
could  choose  whether  or  not  to  review  all  examples. 

The  creation  of  good  examples  usually  depends  on  subject  matter  expenise  and  can 
sometimes  take  a  great  deal  of  time.  The  ID  might  need  to  actively  elicit  such  information  from 
the  SME.  The  SME  will  often  be  the  best  source  about  which  concepts  need  elaboration,  the 
kinds  of  questions  students  ask,  and  the  kinds  of  explanations  that  clarify  most  effectively.  One 
approach  used  is  to  introduce  a  concept,  define  it,  provide  an  example  of  the  concept,  and  then 
an  example  of  what  the  concept  is  not.  Other  examples  may  be  necessary  to  help  the  student 
discriminate  the  new  concept  from  others  learned. 

Recommendation:  Graphics  should  be  used  where  appropriate  throughout  the  lessons. 

Pictures,  illustrations,  and  figures  can  be  motivating  in  a  lesson.  However,  the  graphics 
should  accomplish  specific  instructional  purposes,  such  as  illustrating  critical  points  or  clarifying 
difficult  concepts.  Graphics  should  not  be  used  as  "fillers"  or  they  could  become  distractions  to 
the  lesson  objectives.  Incorporating  a  variety  of  terms  or  concepts  into  a  graphic  can  be  an 
effective  mnemonic  educational  device,  but  accomplishing  this  often  requires  inventiveness  and 
expertise. 

Even  when  a  proficient  graphics  person  is  available,  graphics  require  time  to  create.  A 
screen  of  graphics  will  take  longer  to  create  than  a  screen  of  text.  Our  lessons  might  have  been 
enhanced  with  more  graphics,  such  as  pictures  of  documents  or  copies  of  them  provided  in 
appendices;  however,  time  was  limited  and  other  priorities  took  precedence. 
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Recommendation:  Using  the  right  words  is  essential. 

Selecting  appropriate  terminology  may  be  difficult  when  prospective  learners  represent  a 
broad  range  of  experience  and  knowledge.  An  extensive  glossary  or  mini-dictionary  may  be 
needed  by  the  novice  learner.  When  the  medium  is  printed  materials,  the  reader  needs  to  be 
reminded  periodically  that  a  glossary  is  available.  For  computerized  lessons,  it  is  helpful  if  the 
glossary  can  be  accessible  horn  any  level  of  the  lessons. 

Recommendation:  Students  need  variety  in  lesson  presentation  to  stay  interested  and 
awake. 

First  and  foremost,  infoimadon  must  be  presented  in  a  context  that  the  learners  will 
understand  and  find  useful.  The  learner  should  be  encouraged  to  exercise  cognitive  skills  similar 
to  those  used  in  a  real-world  setting.  Periodically  the  student  should  be  reminded  about  the 
importance  of  the  information  and  how  it  fits  into  the  bigger  picture. 

Some  relevant  application  or  exercise  should  be  interjected  from  time  to  time  to  motivate 
learners  to  use  these  materials.  A  sprinkling  of  humor  can  also  be  effective  in  motivating  the 
student  to  continue. 

Recommendation:  Interaction  between  the  learner  and  the  materials  is  critical  to  the 
learning  process. 

As  the  developers  review  each  page  or  screen,  they  should  ask  questions  such  as: 

1 .  What  does  the  learner  know  so  far? 

2.  How  many  facts  or  concepts  have  been  presented  since  the  learner  last  interacted  with 
the  materials? 

3.  Is  it  time  for  an  exercise  or  an  example? 

4.  How  can  the  user  be  questioned  on  the  content  or  otherwise  made  an  active  participant 
in  the  lesson? 

Recommendation:  Lesson  objectives  should  be  periodically  rechecked. 

As  lessons  are  developed,  lesson  specifications  should  be  periodically  reviewed  to  ensure 
that  the  material  is  clearly  tied  to  the  objectives. 
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Recommendation:  Overall  content  and  presentation  should  be  finalized  as  early  as 
possible. 

This  recommendation  is  especially  important  if  two  versions  of  the  lesson  materials  are 
being  developed.  If  major  content  changes  are  made  after  both  the  versions  are  in  production; 

1.  The  work  is  doubled.  For  example,  bor/i  print  and  computer  versions  must  be  fixed. 

2.  The  chance  of  error  is  increased.  A  change  made  in  the  print  version  might  be 
overlooked  in  the  computer-based  version. 

Recommendation:  If  both  print  and  computer-based  versions  are  to  be  developed,  develop 
the  print  version  first. 

The  method  that  worked  the  best  for  us  was  to  finalize  the  material  in  the  print  version 
before  starting  the  computer-based  version.  We  make  this  recommendation  for  the  following 
reasons; 

1.  Development  of  computer-based  lessons  is  expedited  by  "reading"  (copying)  the 
finalized  content  of  the  print  version  into  prepared  templates  for  computer  storyboards, 
thus  allowing  more  time  for  screen  design  and  incorporation  of  interactivity. 

2.  Working  in  a  word  processor  is  often  simpler  than  working  in  an  authoring  system  for 
developing  computer-based  instruction.  As  mentioned  earlier,  extra  time  must  be 
planned  for  developing  computer-based  lessons  due  to  programming  and  branching 
requirements.  Major  changes  to  lesson  structure  are  often  more  laborious  to  make  to 
computer-based  lessons  than  to  print  lessons.  Also,  all  changes  made  to  computer-based 
lessons  require  added  time  for  the  authoring  system  to  compile  the  revisions,  and  for  the 
author  to  then  look  at  them  on  the  computer  to  see  them  from  the  student's  point  of  view. 

C.  REVIEWING  AND  REVISING 

The  key  to  successful  review  and  revision  is  creating  a  systematic,  standardized  procedure 
for  identifying  and  tracking  all  review/revision  changes. 

Recommendation:  Sufficient  time  should  be  set  aside  for  revision. 

It  is  almost  impossible  to  overestimate  the  time  allowed  for  the  review/revision  process.  No 
matter  how  much  time  is  planned,  it  will  take  longer.  Realize  that  time  must  be  set  aside  for 
transferring  documents  or  computer  disks  from  the  developer  to  the  reviewer  and  back  again  a 
number  of  times  to  ensure  that  all  changes  have  been  examined  and  all  concerns  addressed. 


The  review  and  revision  cycle  can  occur  four  or  more  rimes  before  instrucrional  materials 
are  completed.  Time  must  be  allowed  for  revisions  after  internal  reviews,  after  SME  reviews, 
and  after  both  formative  and  summative  evaluations.  Also,  practical  issues  such  as  fixed 
deadlines  are  always  going  to  be  involved.  During  our  development  effort,  some  of  the 
suggestions  from  the  formative  evaluation,  although  valuable,  could  not  be  acted  upon  because 
of  rime  limitations. 

Recommendation:  A  stopping  point  to  the  revision  cycle  should  be  decided  on  in  advance. 

Although  there  always  seems  to  be  one  more  improvement  that  could  be  made,  schedules 
must  be  met.  In  anticipation,  the  final  revision  process  should  be  set  well  in  advance  of  this  date. 
Alternatively,  the  ID  team  can  decide  in  advance  to  stop  the  process  after  a  certain  number  of 
iterations.  However,  both  of  these  limitations  should  be  overridden  if  the  materials  have  not  met 
quality  standards.  In  other  words,  quality  must  not  be  sacririced,  but  some  reasonable  stopping 
point  must  be  chosen  beyond  which  no  further  changes  will  be  made. 

Recommendation:  The  review/revision  cycle  should  be  structured. 

Before  beginning  the  review  and  revision  process,  a  written  plan  or  flow  chart  should  be 
created  that  schedules  who  reviews  the  materials  first,  second,  third,  and  so  on,  and  how  many 
cycles  there  will  be.  One  possible  review/revision  plan  is  presented  in  Appendix  H. 

Recommendation:  A  standardized  review/revision  form  should  be  designed  for  use  during 
the  review/revision  cycle. 

This  form  can  be  used  by  all  reviewers  to  record  their  suggestions  for  change.  See 
Appendix  I  for  an  example  of  such  a  form.  A  standard  way  to  describe  changes  may  also  be 
specified.  Some  developers  use  proofreader's  marks;  others  develop  their  own  in-house 
conventions.  Such  standards  can  lessen  or  eliminate  the  need  to  ask  one  another,  "What  did  you 
mean  here?"  or  "Where  is  this  error?” 

Following  are  some  suggestions  for  items  that  might  be  included  on  a  review  form; 

1.  Location  of  change.  The  form  could  include  space  for  describing  the  location  of  the 
change,  such  as  the  page  number  (for  a  print  document)  or  the  screen  number  (for  the 
computer-based  version). 

2.  Type  of  change.  A  coding  system  could  be  established  (similar  to  proofreader's 
marks)  to  describe  what  change  to  make. 

3.  Version(s)  to  change.  The  form  could  ask  whether  the  change  is  to  be  made  on  the 
computer  version  or  print  version  or  both.  For  example,  some  simple  format  changes 
will  apply  only  to  one  version.  Content  changes  will  apply  to  both  versions. 
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4.  Date  when  change  is  completed.  The  form  should  provide  space  to  indicate  the  date 
the  change  was  made  and  the  person  who  made  the  change. 

Recommendation:  Changes  between  computer  and  print  versions  must  be  coordinated. 

After  any  reviewer  has  evaluated  either  the  print  or  the  computer  version,  the  comments 
should  be  checked  against  both  versions  of  the  lesson  Major  content  items  should  be  consistent 
between  versions—however,  it  is  up  to  the  ID  team  to  determine  if  it  is  necessary  for  both 
versions  to  be  exactly  the  same,  including  non-crucial  wordings. 


D.  PRODUCING  THE  FINAL  MATERIALS 

Recommendation:  Enough  time  must  be  allowed  for  final  processing  of  materials. 

Although  there  may  be  a  tendency  to  work  up  to  the  last  minute  to  incorporatt  1-  possible 
improvements,  time  for  final  processing  must  be  scheduled.  Making  copies  of  print  documents 
takes  time,  especially  if  the  printing  facility  has  many  customers.  Where  possible,  the  job  should 
be  scheduled  in  advance. 

If  computer-based  lessons  are  being  developed,  time  is  needed  for  making  disk  copies, 
which  may  include  formatting  a  large  number  of  disks.  These  disks  should  be  randomly  tested 
to  ensure  that  they  work  properly.  During  our  effort  a  batch  of  faulty  disks  resulted  in  inoperable 
lessons  that  went  undiscovered  until  the  formative  evaluation. 

The  ID  team  should  make  sure  enough  disks  will  be  on  hand  at  the  critical  ..me.  They  need 
to  be  labeled  consistently,  perhaps  using  color  to  identify  the  disks  that  belong  to  a  specific 
lesson. 

Recommendation:  Coordinators  (especially  those  at  remote  sites)  will  not  generally  want 
to  photocopy  materials.  But  if  time  or  cost  considerations  make  this  necessary, 
coordinators  need  precise  instructions. 

For  example,  coordinators  should  be  instructed  not  to  make  two-sided  copies  if  question  and 
answer  pages  will  end  up  facing  each  other.  Alternatively,  lessons  can  be  designed  so  that  all 
answers  appear  at  the  end  of  the  lessons  rather  than  immediately  following  the  questions. 

Recommendation:  When  materials  are  to  be  mailed,  information  for  the  coordinator 
should  be  placed  on  the  top  of  the  other  materials,  perhaps  in  an  envelope  marked  "READ 
THIS  FIRST." 

This  is  especially  important  when  lesson  materials  have  several  parts  or  sections.  Large 
boxes  filled  with  several  types  of  booklets  and  disks  and  forms  can  discourage  even  the  most 
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willing  coordina.or.  Also,  a  list  of  the  contents  should  accompany  each  batch  of  materials  so 
that  the  coordinator  knows  everything  has  arrived. 


VI.  EVALUATION  PHASE 

In  the  evaluation  phase  of  the  SE/ISD  process,  the  courseware  developed  is  assessed  by 
conducting  both  formative  and  summative  evaluations. 


A.  FORMATIVE  EVALUATION 

In  some  sense,  the  formative  evaluation  might  be  considered  as  pan  of  the  development 
phase,  since  the  results  of  this  evaluation  are  .ised  to  further  revise  the  materials  to  improve  the 
effectiveness  of  instruction.  The  formative  evaluation  is  conducted  to  discover  both  minor  and 
major  problems  in  the  materials  and  to  find  out  how  well  the  lesson  materials  work.  The 
instructional  materials  are  presented  to  representative  members  of  the  target  population  in  a 
setting  that  closely  resembles  the  future  learning  situation.  The  learners  serve  as  trial  users  of 
the  materials  to  uncover  any  problems  that  might  still  exist.  For  example,  do  the  learners  have 
any  special  problems  in  following  the  lesson  structure?  Are  the  ideas  expressed  clearly?  How 
long  does  it  take  to  complete  a  lesson?  Did  the  learners  have  any  content  questions  after 
completing  the  lesson? 

The  following  discussion  is  based  on  our  evaluation  method,  whk  h  involved  gathering  data 
through  feedback  survey  forms  from  evaluators  in  several  cities.  Our  lessons  learned  focus  on 
preparing  for  the  evaluation,  tracking  the  evaluation,  and  evaluating  feedback. 

1.  Preparing  for  the  Evaluation 

Recommendation:  Objectives  of  the  evaluation  must  be  defined. 

There  are  likely  to  be  questions  asked  about  the  materials  that  interest  the  ED  team  more 
than  others.  V/hether  the  evaluation  is  conducted  in  person,  in  groups,  or  from  a  remote  site,  the 
ID  team  members  must  be  clear  in  their  own  minds  about  the  kinds  of  information  that  need  to 
gathered.  They  should  concentrate  on  qu<“*:tions  that  will  elicit  answers  useful  for  revising  the 
lessons.  For  example,  participants  should  not  be  asked  if  they  would  rather  not  le?:.n  about  a 
particular  subject  if  they  have  no  choice  in  the  matter. 

The  ID  team  should  anticipate  what  answers  are  likely  and  consider  what  changes  might 
have  to  be  rr?de  based  on  those  answers.  Open-ended  questions,  multiple  choice,  and  survey 
questions  that  ask  evaluators  to  choose  a  point  on  a  continuum  are  all  useful.  Questions  that 
force  evaluators  to  make  a  choice  are  preferred  over  those  that  permit  them  to  select  some 
middle  ground  positioti. 
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Only  une  or  two  general  "How  did  you  like  it?"  questions  should  be  asked.  General  testing 
procedures  should  be  followed,  such  as  mixing  the  questions  so  that  participants  don't  form  a 
response  set  (such  as  choosing  all  I’s  or  5’s  or  lO’s  on  a  10-point  continuum). 

Recommendation:  A  plan  should  be  established  for  dealing  with  the  feedback  results. 

1  he  plan  should  addresc  who  receives  the  feedback  results,  the  amount  of  time  set  aside  for 
revision,  the  number  of  comments  of  a  certain  type  that  should  warrant  a  change,  etc.  When  the 
time  comes  to  actually  implement  these  plans,  these  early  decisions  may  be  modified  or 
reversed,  but  the  effort  w^  s  not  wasted.  Thinking  about  these  issues  helps  shape  the  evaluation 
and  can  even  affect  the  kinds  of  questions  asked  during  the  evaluation. 

Recommendation:  When  preparing  materials  for  evaluation,  the  job  of  the  site 

coordinator  should  be  made  as  simple  as  possible. 

A  coordinator  in  this  case  is  someone  (perhaps  at  a  remote  site)  who  has  volunteered  to  find 
students/evaluators  willing  to  use  the  new  training  materials  in  a  systematic  way  and  to  provide 
feedback  on  these  materials.  The  coordinator  probably  already  has  a  full  schedule  and  does  not 
have  time  to  take  care  of  unanticipated  problems.  The  training/testing  site  situation  should  be 
considered  ahead  of  time  to  anticipate  both  student  and  coordinator  needs  and  questions.  These 
needs  should  be  met  in  the  materials  provided. 

a.  WrUten  Instructions 

The  rj  team  should  prepare  instructions  fo:  snidents/evaluator-'  that  describe  the  purpose  of 
the  ev?.iuation  and  tells  them  what  to  do.  This  will  save  coordinator  time  and  provide  a 
somewhat  standardized  overview  of  the  effort  to  all  participants. 

b.  Evaluation  Forms  for  Interviewers  ?  Evaluators 

A  sample  of  instructions  to  help  coordinators  organize  their  efforts  is  included  in  Appendix 
J,  and  a  sample  of  an  evaluation  form  we  used  is  provided  in  Appendix  K.  The  form  provides 
opportunities  for  both  op^n-ended  and  forced-choice  responses.  This  option  takes  into  account 
individual  differences  in  response  style. 

Recommendation:  An  eariy  deadline  should  be  set  for  return  of  materials. 

The  deadline  set  should  be  one  that  gives  evaluators  a  reasonable  amount  of  time  to  do  the 
evaluation  but  yet  is  actually  earlier  than  required.  Even  the  most  willing  coordinators  are  likely 
to  be  late  since  they  are  dealing,  in  turn,  with  people  who  may  be  late  completing  the  lessons. 
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2.  Conducting  and  Monitoring  the  Evaluation 

Recommendation:  Careful  records  should  be  kept  of  contacts  with  coordinators  from 
remote  sites. 

All  attempts  to  contact  coordinators  should  be  recorded.  This  information  is  useful  in 
reviewing  the  whole  process  and  can  provide  the  basis  for  commendation  of  coordinators  for 
exemplary  effons. 

If  the  evaluation  takes  place  over  a  period  of  weeks  or  months  and  if  the  students  are 
working,  per  direction,  at  their  own  pace,  weekly  or  biweekly  calls  should  be  made  to  all 
evaluation  sites  to  inquire  about  any  possible  problems.  If  time  and  resources  permit,  evaluation 
sites  should  be  visited  to  observe  students  at  work,  to  compare  the  setups  at  the  various  sites,  and 
even  to  interview  some  of  the  evaluators. 

3.  Evaluating  Feedback 

Recommendation:  Sufficient  time  must  be  set  aside  for  revisions  based  on  the  evaluation 
feedback. 

Recommendation:  SMEs  should  be  involved  in  evaluating  feedback. 

We  received  some  contradictory  opinions  about  the  same  lessons.  One  evaluator  thought  a 
particular  lesson  was  clear,  concise,  and  useful,  while  another  thought  it  was  vague,  confusing 
and  a  waste  of  time.  It  is  difficult  to  determine  which  comments  to  heed.  SMEs  are  particularly 
helpful  at  this  stage  in  trying  to  determine  how  to  revise  the  instructional  materials,  at  least 
regarding  content  issues. 

Determining  what  feedback  to  incorporate  into  the  lessons  often  depends  on  the  time 
available  for  revisions  and  the  quality  standards  set  for  the  lessons.  Hf  course,  obvious  content 
errors,  computation  errors,  and  repeated  comments  about  unclear  concepts  should  be  addressed. 
But  most  other  lesson  elements  are  up  to  the  discretion  of  the  development  team. 


B.  SUMMATIVE  EVALUATION 

In  a  summative  evaluation,  the  concern  is  for  validating  the  materials.  This  means,  is  the 
student  performance  and  achievement  on  the  new  materials  comparable  or  superior  to  that  on  the 
old?  Summative  evaluations  usually  involve  a  larger  number  of  students  than  the  formative 
evaluation,  and  include  formal  testing  procedures,  control  and  experimental  groups,  etc. 
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Recommendation:  Coordinators/facilitators  and  the  students  should  be  provided  with  the 
same  sorts  of  aids  and  forms  as  provided  during  the  formative  evaluation. 

The  coordinator  forms  and  evaluator  feedback  forms  described  in  the  formative  evaluation 
can  be  revised,  if  necessary,  and  used  for  the  summative  evaluation  as  well.  Since  the  objectives 
of  the  summative  evaluation  are  somewhat  different,  questions  and  instructors  may  change,  but 
the  basic  intent  of  both  sets  of  forms  is  the  same-‘to  simplify  the  coordinator’s  effons  and  elicit 
feedback  from  the  students. 

Recommendation:  If  possible,  evaluation  sites  should  be  chosen  that  offer  some  sort  of 
motivator  to  students  and  other  reviewers/evaluators  so  that  they  are  eager  to  cooperate. 

Evaluators  need  some  motivation  to  participate— whether  it  be  course  credit  or  recognition  of 
some  sort,  such  as  a  certificate  of  achievement.  The  sites  chosen  should  be  encouraged  to 
develop  a  motivational  system  if  one  is  lacking.  It  is  unlikely  that  students  will  spend  more  than 
a  few  short  hours  actively  learning  material  that  will  serve  no  useful  purpose  to  them. 

Recommendation:  When  comparing  a  transportable  course  with  a  resident  one,  an 
agreement  with  the  resident  institution  should  be  set  up  so  that  the  final  evaluation  test  is 
administered  to  a  sufficient  number  of  students. 

Resident  teaching  institutions  often  experience  changes  in  personnel,  in  management 
direction,  and  in  course  schedules.  A  written  agreement  is,  therefore,  important  to  guarantee  that 
students  can  be  tested  with  a  panicular  instrument. 

Recommendation:  Adequate  information  about  the  learner's  past  educational  and  job 
experience  should  be  collected. 

The  learning  settings  for  resident  courses  and  transportable  ones  vary  radically.  Learners 
who  use  transponable  materials  will  often  be  able  to  use  them  at  any  time  of  day  and  under 
diverse  conditions.  The  learner's  past  educational  and  job  experience  might  be  the  one  variable 
that  must  be  considered  when  evaluating  final  performance  data  between  the  two  groups.  At  the 
least,  learners  with  less  relevant  education  and  experience  can  be  expected  to  take  longer  to 
complete  the  lessons. 

Recommendation:  To  offset  the  impact  of  attrition,  at  least  20  percent  more  students  than 
needed  should  be  scheduled  to  test  the  materials. 

Despite  all  good  intentions,  a  number  of  the  students  who  originally  agree  to  test  the 
instructional  materials  will  find  it  impossible  to  begin  or  to  complete  the  process.  This  happens 
more  frequently  when  lessons  are  to  be  completed  over  a  period  of  several  sessions  or  days.  If 
the  sample  is  larger  than  needed,  the  impact  of  these  dropouts  on  evaluation  will  be  negligible. 
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VII.  CONCLUSIONS 


This  "lessons  learned"  document  is  organized  STOuiiu  it.  ''mmendations  tnat  grew  out  of  the 
NAVPERSRANDCEN  experience  with  print  and  computer-based  media.  The  reader  is  directed 
to  the  selected  references  in  Appendix  B  for  guidance  on  how  to  work  effectively  with  other 
media  such  as  interactive  video. 

Our  experience  makes  clear  that  the  development  of  transportable  instructional  materials  is  a 
complex  process  requiring  extensive  planning.  A  team  effort  is  required  to  accomplish  the  many 
tasks  at  hand.  To  do  them  all  successfully,  ID  team  members  must  collectively  represent  a  wide 
range  of  skills. 

Transportable  courseware  makes  good  sense  any  time,  but  as  funding  within  the  Department 
of  Defense  continues  to  shrink  in  response  to  changing  world  conditions,  it  makes  even  nrwre 
sense.  Transportable  materials  reduce  the  need  for  residential  classrooms  and  a  large  faculty. 
They  permit  education  to  go  on  in  any  environment  and  for  larger  numbers  of  people.  No  longer 
should  students  have  to  leave  their  work  place  to  travel  long  distances  to  receive  training.  And 
instructors  have  the  best  of  both  worlds.  Now  they  have  the  ability  to  standardize  education  and 
at  the  same  time  adapt  it  to  meet  individual  needs. 
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SYSTEMS  ENGINEERING  FOR  INSTRUCTIONAL  DEVELOPMENT  (SEID) 

PROCESS 


This  appendix  presents  a  brief  description  of  the  phases  in  the  Systems  Engineering  for 
Instructional  Development  (SEID)  process.  The  literature  cited  in  Appendix  B  provides  in-depth 
discussions  of  the  SEID  process  and  the  more  generic  Instructional  Systems  Development  (ISD) 
process. 

1.  Analysis 

The  goal  of  the  analysis  phase  is  to  determine  what  and  who  needs  to  be  taught.  During  this 
phase,  analysts  review  the  current  operations  of  a  job  or  learning  task  to  determine  if  there  is  an 
instructional  problem.  If  it  is  determined  that  improved  training  is  needed,  the  focus  shifts  to  the 
specific  tasks  that  need  to  be  trained. 

The  learner  population  is  also  assessed  to  determine  who  they  will  be,  what  they  already 
know,  and  what  they  need  to  know.  In  addition,  the  learning  situation  (e.g.,  job  site)  is  examined 
to  assess  its  conditions  and  the  resources  available.  Management  issues  are  also  considered, 
including  management  tasks,  responsibilities,  and  time  lines.  Based  on  these  reviews,  analysts 
determine  what  needs  to  be  changed  in  the  existing  education  or  training  program.  The  goals, 
constraints,  and  priorities  of  the  instructional  program  or  products  are  then  established.  The 
products  of  the  analysis  phase  generally  consist  of  task  descriptions,  course  learning  objectives, 
and  course  prerequisites. 


2.  Design 

Once  the  analysts  have  determined  what  instruction  is  needed,  the  goal  of  the  design  phase 
is  to  determine  how  the  instruction  will  be  performed.  The  goals,  constraints,  and  priorities 
developed  in  the  analysis  phase  are  clarified  and  made  specific. 

The  designer's  role  is  to  determine  the  most  appropriate  media  for  instruction,  methods  of 
instruction,  specific  content  of  instruction,  and  time  estimates.  The  designer  must  also  deteimine 
instructional  strategies  (including  response  and  feedback  strategies  as  well  as  sequence  of 
presentation),  assessment  strategies,  and  management  strategies  for  collecting,  reporting,  and 
using  the  collected  data. 

The  final  products  of  the  design  phase  are  lesson  specifications  that  document  the 
presentation  and  evaluation  strategies.  These  specifications  include  prerequisite  learning,  lesson 
structures,  presentation  strategies,  examples,  evaluation  strategies,  and  reviewer  guidelines. 
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They  also  provide  detailed  instructional  objecdves,  with  test  items  written  to  match  these 
objecdves.  The  specificadons  provide  the  lesson  authors,  graphic  ardsts,  and  other  instrucdonal 
design  specialists  with  the  informadon  needed  to  develop  the  final  learning  materials.  These 
specificadons  are  designed  at  a  very  detailed  level  to  ensure  that  lesson  authors  do  not  have  to  be 
subject  matter  experts. 


3.  Development 

After  the  analysts  and  designers  have  defined  and  outlined  what  will  be  taught  and  how  and 
when  and  why,  the  instrucdonal  authors  create  the  actual  instrucdonal  materials.  All  parts  of  the 
materials  are  based  on  the  media,  methods,  and  instrucdonal  strategies  oudined  in  the  lesson 
specificadons.  It  is  during  the  development  phase  that  scripts  and  storyboards  are  written, 
graphics  and  other  pictorial  illustradons  are  created,  and  lesson  materials  and  programming  for 
computer-based  courseware  constructed.  If  any  flaws  or  omissions  occurred  during  the  analysis 
and  design  phases,  they  will  show  up  at  this  phase.  Making  correcdons  to  analysis  and  design 
problems  can  delay  the  compledon  of  this  phase  of  the  process. 


4.  Implementation 

During  the  implementation  phase,  the  instrucdonal  materials  are  presented  to  students  and 
instructors  in  the  field.  The  goal  of  this  phase  is  to  ensure  that  a  valid  learning  program  is 
established,  maintained,  and  improved  over  dme.  This  is  accomplished  by:  (a)  examining 
implementation  performance  data  and  experiences,  (b)  incorporating  anecdotal  instructor 
experiences  that  are  encountered,  and  (c)  incoiporadng  feedback  from  learners  and  their  job 
supervisors. 


5.  Evaluation 

The  evaluadon  phase  involves  conducdng  the  formadve  and  summadve  evaluadons  of  the 
instrucdonal  materials. 

The  purpose  of  a  formadve  evaluadon  is  to  gather  informadon  about  the  readability  and 
understandability  of  the  material,  the  ease  of  use,  instrucdonal  goodness,  and  so  on.  A  formadve 
evaluadon  can  involve  a  small  sample  of  people,  such  as  students  who  represent  the  target 
population,  or  somedmes  only  subject  matter  experts.  Although  potendal  students  are  an 
excellent  source  of  feedback,  somediiKS  the  formadve  evaluadon  is  conducted  instead  with 
experienced  teachers  or  job-holders-another  kind  of  subject  matter  expert  in  the  field. 
Formadve  evaluadon  involves  either  small-group  or  full-group  operadonal  trials  of  the 
instrucdonal  materials.  This  phase  of  evaluation  is  often  conducted  BEFORE  the  courseware  is 
implemented,  so  that  the  revisions  based  on  this  evaluadon  can  be  included  in  the  fielded 
lessons. 
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Formative  evaluations  can  consist  of  individual  interviews,  group  interviews,  or  surveys 
(sending  feedback  forms  to  remote  sites  for  students  to  complete).  Results  from  all  of  these 
methods  have  been  shown  to  be  comparable,  though  they  differ  in  cost,  with  individual 
interviews  being  the  most  expensive  in  terms  of  time  and  personnel  resources  and  the 
survey/feedback  foim  method  being  the  least  expensive. 

Summadve  evaluation  consists  of  evaluation  of  peifonnance  data  and  feedback  from 
graduates  of  the  courseware  as  well  as  development  of  "lessons  learned"  by  the  ID  team.  The 
goal  of  the  summadve  evaluadon  is  to  determine  the  value  or  worth  of  the  instruedon. 
Performance  data  typically  include  the  results  of  criterion-referenced  assessment  instruments  that 
measure  student  mastery  of  lesson  objeedves.  The  evaluators,  along  with  the  subject  matter 
expens,  review  the  performance  data  and  the  feedback  and  determine  if  the  materials  are 
adequately  communicadng  what  needs  to  be  taught. 

Whereas  the  formative  evaluadon  is  expressly  conducted  to  gather  data  to  revise  the 
materials,  the  summadve  evaluadon  is  conducted  to  determine  the  value  of  the  materials  for  a 
pardcular  subject  population  or  a  pardcular  learning  setdng,  or  both.  The  analysis  includes  how 
well  the  content  has  been  covered,  how  well  the  objeedves  of  the  lessons  have  been  met,  and  the 
relationship  between  the  test  instruments  and  the  objeedves.  All  of  these  evaluadons  can  trigger 
revisions  in  course  presentation  or  evaluadon. 
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LESSON  SPECIFICATION 

LESSON  NUMBER  01 

LESSON  TITLE  Introduction  to  Funds  Management  in  the 
Department  of  Defense 

HOURS  OF  INSTRUCTION  Transportable  -  3  hours 

MEDIA  Self-Paced  Print  and  Computer-Assisted  Learning  (CAL) 

PREREQUISITES 

The  learner  should  have  completed  DAEP  Lessons  1  through  6  and  TPM 
Lessons  1  through  3  before  starting  this  lesson. 

LESSON  MAP  This  is  the  first  lesson  in  a  series  of  six  REM  lessons. 
REM  01  is  a  three  hour  lesson  composed  of  six  segments  to  be  taken  in 


sequence  as  shown  below: 

Segment  1  The  Historical  Legal  Basis  of  Federal 

Budgeting.  .25  hr. 

Segment  2  Monetary  Concepts  Related  to  Budgeting  .50  hr. 

Segment  3  Methods  and  Techniques  of  Cost  Estimating  .50  hr. 

Segment  4  Learning  Curve  1.0  hr. 

Segment  5  Life  Cycle  Cost  Management  and  Design  to 

Cost  Management  in  Relation  to  Cost 
Estimating/Budgeting  .50  hr. 

Segment  6  Management  Reserve  .25  hr. 

LESSON  INTRODUCTION 


The  funding  of  defense  systems  acquisition  has  its  basis  in  the 
Constitution  and  subsequent  laws  enacted  by  Congress.  Funds  management 
is  a  continuous  process  within  the  Program  Office.  Budgeting  data  is 
used  for  budget  submissions  for  funding  and  for  continuous  “reality 
checking"  within  pro^ams.  Cost  estimations  provide  the  basis  for 
making  budgeting  decisions. 

LESSON  OBJECTIVES 

1.  Briefly  explain  the  historical  legal  basis  of  Federal  budgeting. 
(3.1.1) 

2.  Identify  definitions  of  monetary  concepts  related  to  budgeting  and 
their 'sequence:  Budget  Authority,  Commitment,  Certification, 
Obligation,  Cost  Incurrence,  "Expenditure,  Outlay.  (3.1.2) 

[Note:  This  is  a  brief  introductory  segment  to  introduce  these 
terms;  additional  information  will  be  taught  in  REM  05) 
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3.  Identify  explanations  of  the  following  methods  and  techniques  of 
cost  estimating  and  the  primary  use  of  each:  Analogy,  Parametric, 
Engineering,  Extrapolation  from  Actuals,  and  Price  Lists.  (3.1.3) 


4. 


Given  appropriate  data,  determine  learning  car/e  lot  midpoints 
using  graph  (curved  line  and  log- log)  solutions.  (3  1.4) 


5.  Explain  Life  Cycle  Cost  Management  and  Design  to  Cost  Management  in 
relation  to  cost  estimating/budgeting.  (3.1.5) 


6  Explain  the  term  "management  reserve" ,  including  its  need  in  cost 
estimates .  (3.1.6) 

COMMON  ERRORS 


1-  The  concepts  (Commitment,  Ce.tificatior ,  and  Obligation)  are  often 
confused.  (Segment  01.2) 

2.  The  concepts  Expenditure  and  Outlay  are  commonly  used 
interchangeably.  They  are  also  commonly  misused  in  the  "field". 
(Segment  01.2) 

3.  The  learner  will  often  want  to  use  just  one  cost  estimating  method, 
when  in  actuality  all  could  be  used  to  determine  estimate 
correlation.  (Segme.nt  01.3) 

ITEM  PRODUCTION 

Practice  items  and  test  items  will  cor'i.,t  of  fill-in-the-blank, 
true/false,  short  answer,  and  listing  t^pes  of  items  formatted  to 
multiple  choice  questions. .  Example  items  will  be  constructed  in 
support  of  the  segment  geneit ’ities.  The  number  of  example, 
practice,  and  test  items  for  each  segment  are  shown  in  the  matrix 
below. 


SEGMQITS 


EXAMPLES 


r 

Segment  1  | 

N/A  1 

1 

1 

2  1 

1 

Segment  2  1 

N/A  1 

2 

1 

4  1 

1 

Segr  t  3  | 

N/A  1 

4 

1 

8  ( 

1 

Segment  4  | 

2  1 

3 

1 

6  1 

1 

Segment  5  | 

N/A  1 

3 

1 

6  1 

1 

Segment  6  | 

N/A  1 

1 

1 

2  1 

IT^ 

PRACTICE 


TEST 
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LESSON  AUTHOR  REFERE3ICES 

Teach ir.c  More" 

-  Teaching  Note:  Cost  bstimating  Methodologies  (Jan  88) 

-  Teaching  Note:  Parametric  Cost  Analysis  (Jan  88) 

-  Teaching  Note:  Cost  Quantity  Relationships  “Learning  Curve 
Theory  (Jan  88) 

-  Teaching  Note:  Life  Cycle  Cost  Management  (Jan  88) 

-  Teaching  Note:  President's  Budget  Submission,  Congressional 
Authorization  and  Appropriation  [from  REM  04]  (Jan  88) 

DoD  Directives/Instructiors/Standards 

-  DoDI  5000.33  -  Uniform  Budget/Cost  Terms  and  Definitions 
(15  Aug  77) 

-  DoDD  4245.3  -  Design  to  Cost  (6  Apr  83) 

Articles /Books 

-  Troy  Caver,  "Life  Cycle  Cost:  Attitudes  and  Latitudes",  Defense 
Manacement  Journal.  July-August  1979. 

-  J.  Witt,  "Life  Cycle  Cost  Analysis",  Technical  Perspective. 

May  1974. 

-  J.  Underwood  and  B.  Retterer,  “Design  to  Cost“,  Technical 
Perspectives" .  October  1976. 

-  A  Glossary  of  Terms  Used  In  The  Federal  Budget  Process. 

U.S.  General  Accounting  Off3.ce,  March  1981. 

Example  Documents 

-  Prototype  Lesson  BREM  09  (The  Prototype  Lesson  BRIM  09  was 
developed  in  1987  under  a  previous  contract  by  Instructional 
Science  and  Development,  Inc.  (ISD)  for  the  Navy  Personnel 
Research  and  Development  Center  (N?RD(")  and  the  Defense  Systems 
Manage.me.nt  College  (DSMC).) 
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SEGMENT  SPECIFICATION 


SEGMENT  NUMBER  REM  01.1 

SEGMENT  TITLE  The  Historical  Legal  Basis  of  Federal 
Budgeting 

SEGMENT  OBJECTIVE 

1.  Briefly  exnlain  the  historical  legal  basis  of  Federal  budcetinc. 

(3.1.1) 

OBJECTIVE  TYPE 

1 .  Re.’ne.n±)er/Concept 

GENERALITY 

1.  The  following  legal  documents  provide  the  legal  basis  of  Federal 

Budgeting; 

-  The  Constitution  provides  t.hat  financial  policy  is  the 
responsibility  of  Congress.  Article  I,  Section  9  states,  "No 
money  shall  be  drawn  from  the  Treasury,  but  in  Consequence  of 
Appropriations  made  by  law. . . " .  Appropriations  are  a  part  of  an 
Appropriation  Act  or  an  act  of  Congress  that  provides  a  specified 
amount  of  budget  aut.hority  (authority  to  enter  into  obligations 
which  generally  result  in  Lmmediate  or  future  disburse.ments  of 
Government  funds ) . 

-  Budcetinc  and  Accounting  Act  ( 1921 ^  set  up  the  requirements  by 
which  t.he  Executive  Branch  prepares  and  submits  the  budget  to 
Congress. 

-  Congressional  Budcet  and  Impoundment  Control  Act  ( 19*^4 ^  c.hanged 
the  fiscal  year,  required  t.nat  Congress  look  at  spending  and 
reve.nues,  and  established  a  timetable  for  budgeting  activities. 

-  Gramm-Rudman-Hol lines  Act  f 1985 1 .  as  revised,  established  an 
automatic  deficit  reduction  mechanism,  revised  the  budgeting 
timetable,  and  streamlined  the  process  of  passing  the  budget. 

ITEM  PRODUCTION 


1  1  ITEMS  1 

1  OBJECTIVES  1 

EXAMPLES 

1  PRACTICE  1 

TEST  1 

1  Objective  1  1 

N/A 

1  ^  1 

2  1 
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EXAMPLE/HELPS 


1.  Refer  back  to  DAEP  03  where  laws  affecting  the  federal  acquisition 
process  were  learned.  The  laws  in  this  lesson  (REM  01.1)  affect 
the  financing  of  federal  acquisitions. 

SAMPLE  PRACTICE  ITEM 

1.  The  act  that  established  a  "deficit  reduction  mechanism"  was  the 


a.  Budgeting  and  Accounting  Act 

b.  Congressional  Budget  and  Impoundment  Control  Act 

*  c.  Gramm-Rudman-Hollings  Act 

d.  Armed  Services  Procurement  Act. 

SAMPLE  TEST  ITEM 

1.  The  Constitution  states,  "No  money  shall  be  drawn  from  the 
Treasury,  but  in  Consequence  of  _ ..." 

a.  Authorizations 

b .  Budget  Authority 

*  c .  Appropriations 
d.  Allocations 

GRAPHIC  SPECIFICATIONS 

1.  Introduction  graphic:  Constitution 

2.  Progressive  disclosure  of  1)  Constitution,  2)  1921,  3)  1974, 
and  1985 
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APPENDIX  D 

SAMPLE  DIRECTORY  STRUCTURE  FOR  ORGANIZING  COMPUTER  FILES 
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APPENDIX  E 

STRUCTURE  AND  NAMING  CONVENTIONS  FOR  PRINT,  COMPUTER-BASED, 
AND  GRAPHICS  FILES  FOR  THE  FUNDS  MANAGEMENT  (FM)  LESSONS 
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EXAMPLE  STRUCTURE  AMD  MAMIMO  C0MVEMTI0M8 


DIRECTORY  STRUCTURE  AMD  NAMINO  COMVENTIONB 

Directory  structure  should  reflect  the  hierarchy  of  your 
courseware  structure. 


Directory  Level  l:  Name  top  level  directory  the  name  of  the 

course :  FM 


Example  c:\FM 

Directory  Level  2:  Name  the  second  level  directories  the  name  of 

the  lesson:  FMl,  FM2,  FM3,  FM4,  FM5 

Example  c : \FM\FM1 

Directory  Level  3:  The  name  of  the  directories  corresponding  to 

areas  within  or  version  of  a  lesson. 


KEY: 

X  =  lesson  name 

Y  =  segment  number 

1. 

FMXPT 

Directory  for  Print  Version 

2. 

FMXUG 

CAL  User's  Guide  Directory 

3. 

FMXSA 

CAL  Introductory  Information 

Directory 

4. 

FMXSB 

CAL  Function  Keys  Directory 

5. 

FMXSY 

CAL  Segment  Directories 

Example 

C:\FM\FM1\FM1UG  (directory 
for  the  FMl  CAL  User's  Guide) 
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The  following  naming  conventions  are  examples  used  from  the  Funds 
Management  (FM)  lessons. 

Print  Version  Files 


KEY 

X  lesson  number 

Y  letter  (A,  B,  C,  etc.)  indicating  file 

sequence 

Z  segment  number  or  letter  for  working  files 


Examples 

FMX.PT  Name  of  final  print  version  file,  if  only  one  file 
is  needed 

Example  FMl.PT  FM  lesson  1  print  version  file 

PMXY.PT  Name  of  final  print  version  files,  if  one  file 
cannot  fit  onto  a  floppy  disk,  where  Y  =  A,  B, 
etc.  indicating  lesson  part. 

Example  FMIA.PT  FM  lesson  1  print  version 

file,  part  A  (file  does  not 
contain  the  entire  lesson  1) 

FMZ8Z.pt  Name  of  working  files  for  segments  (optional) 

Example  FMISI.PT  FM  lesson  1  print  version 

working  file  for  segment  1. 
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CAL  Version  Fil«s 


KEY 

§  functional  area 

#  lesson  number 

$  segment  number 

%  current  menu  selection  number  (cmsn) 

?  page  number,  general  number 


directory  for  FM  lessons 

directory  for  a  specific  FM  lesson 
(example:  FM3  for  FM  lesson  3) 


FM  directory  top  level 
FM#  directory 

build.bat 

qbuild.bat 

distrib.bat 

cvarsfi 

FMfSA  directory 

initial 

intro 

calop 

lesorg 

lesio 

pre 

post 

build.bat 

qbuild.bat 


batch  file  for  compiling 
storyboards 

"quick”  batch  file  for 
compiling  segment  files 

batch  file  for  creating  lesson 
distribution  disks 

file  defining  variables  used 
for  a  lesson  (for  authoring 
system) : 

For  Lesson  FM  3,  variables 
would  be  defined  in  file 
"cvarsf3" 


file  for  initializing 
variables 

lesson  introductory  graphics 
CAL  operating  procedures 
lesson  organization 
information 

lesson  introduction  and 

objectives 

pre-quiz 

post-quiz 

batch  file  for  compiling 
storyboards 

"quick"  batch  file  for 
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compiling  segment  files 

*.pic 

all  graphic  files  with  .pic 
extension  for  the  files  in  the 

directory:  •**”  is  wild  card 
symbol 

*.cut 

all  graphic  files  with  .cut 
extension  for  the  files  in  the 

directory:  "*'•  is  wild  card 
symbol 

FM#SB  directory 

calfl 

calf2 

calf3 

calf4 

roalnmenu 

calf6 

calf? 

calfS 

calf9 

calflO 

build.bat 

qbuild.bat 

*,pic 


*.cut 


function  key  1 
function  key  2 
function  key  3 
function  key  4 
main  menu  function  key 
function  key  6 
function  key  7 
function  key  8 
function  key  9 
function  key  10 
batch  file 
batch  file 

graphic  files  with  .pic 
extension  for  files  in  the 
directory 

graphic  files  with  .cut 
extension  for  files  in  the 
directory 


FM#S$  directory 


§#$in 

segment  menu:  f31m  (segment 
menu  for  lesson  FM  3,  segment 
1) 

segment  introduction  and 
objectives 

e#$io 

§#$ci 

segment  "Core”  information 

§#$mi 

segment  "More"  information 

§#$e 

segment  "Example" 

e#$sl 

segment  "Summary,"  one  screen 

e#$s2 

segment  "Summary,"  more  than 
one  screen 

§#$pl 

segment  "Practice,"  one  item 

§#$p2 

segment  "Practice,"  more  than 
one  item 

build.bat 

batch  file 

qbuild.bat 

"quick"  batch  file 
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*.pic  graphic  files  with  .pic 

extension  for  directory 
*.cut  graphic  files  with  .cut 

extension  for  directory 

Graphics  Files 

The  graphics  also  need  to  be  named  in  a  logical  manner, 
especially  if  they  are  to  be  imported  from  an  external  graphics 
program.  Often,  the  graphic  will  exist  with  two  different  names 
—  and  the  extension  can  sometimes  be  the  feature  that  can 
distinguish  between  the  graphics  program  version  and  the  word 
processor  or  authoring  system  version.  Special  requirements  need 
to  be  considered  when  including  a  graphic  in  either  a  word 
processor  or  in  an  authoring  program. 

The  examples  below  present  templates  for  file  names  of  Dr.  Halo 
III  graphics,  and  the  Word  Perfect  graphic  converted  from  the  Dr. 
Halo  graphic  for  print  versions. 


KEY 

@  functional  area 

#  lesson  number 

$  segment  number 

%  current  menu  selection  number  (cmsn) 

?  graphic  number,  general  number 

x  lesson  sections,  where  x  can  be: 

i  Introduction  and  Objectives 

c  Core  information 

m  More  information 

e  Examples 

s  Summary 

p  Practice  items  and  feedback 
q  Quizzes  (pre  and  post) 

.pic  extension  for  Dr.  Halo  full  screen  graphic 
.cut  extension  for  Dr.  Halo  partial  screen  graphic 
.wpg  extension  for  Word  Perfect  graphic  (converted 
from  the  Dr.  Halo  .pic  graphic 


Q#$x?  Example!  F31C2.pic,  F31C2.cut,  and/or  F31C2.wpg 

where:  F  =  an  FM  lesson 

3  =  Lesson  3 

1  =  Segment  1 

C  =  for  Core  information 

2  =  graphic  2  in  the  Core  information 
file 

F31C2.pic  is  a  Dr.  Halo  full  screen  graphic 

F31C2.cut  is  a  Dr.  Halo  cut  graphic 

F3lC2.irpg  is  a  the  Word  Perfect  graphic  converted 
from  the  Dr.  Halo  .pic  graphic. 
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APPENDIX  F 

SAMPLE  FORMAT  STANDARDS  FOR  DEVELOPMENT  OF  PRINT  LESSONS 
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FORMAT  GUIDELINES 


Margins 

Page  Numbering 
Footer  A 
Footer  B 

Indent 

Tab 

Fonts 

Headings 


PRINT  AND  CAL  USER’S  GUIDE  VERSIONS 

v1  24  May  1989 


Preset 

Preset 

Set  to  FM  0X0.  Change  to  appropriate  X. 

Generic  footers  preset.  Change  Segment  footer  to  appropriate 
X  on  page  1. 

Preset.  Use  indent  rather  than  tab  for  'hanging'  info,  e.g., 
questions,  lists,  glossary,  etc.  Change  tabs  only  for  special 
cases,  e.g.,  table,  then  change  back  to  original. 

Preset.  Change  tabs  only  for  special  cases,  e.g.,  table,  then 
change  back  to  original. 

Helv  12  is  the  base  font.  Only  first  order  headings  are  Heiv  14 
?t- 

Suggestion:  To  bold  titles,  words,  etc.  use  F6.  then  you  don’t 
have  to  use  Ctri  F8  to  change  fonts  and  then  change  back. 

Exception:  Lesson  and  Segment  objectives  have  been  preset 
to  Helv  12  bold. 

First  order 

Helv  14  bold,  caps,  centered,  top  of  page 
6  HRTs  after.  0  HRTs  before 
Second  order 

Helv  12,  bold  (use  F6),  caps,  centered,  top  of  page 
(e.g..  Core  Information)  unless  preset  (e.g..  Module 
Components) 

3  HRTs  after,  4  HRTs  before  (unless  after  first  order, 
then  6  HRTs) 

Third  order 

helv  12,  bold  (F6),  caps,  left  "istified 
3  HRTs  after,  3  HRTs  before 

Fourth  order 

Helv  12,  bold  (F6),  nc  caps,  left  justified 
2  HRTs  after,  3  HRTs  before 
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X  Wild  Card  X  denotes  place  for  you  to  substitute  appropriate  ir(o,  or 

add/delete  info. 

Segment  X  Use  Segment  1  template  for  other  segments.  Change  footer 

A.  Select  appropriate  What  To  Do  Next  sentence  at  end  of 
segment. 

Table  of  Contents 

First,  second,  and  third  order  headings  should  be  listed  m 
Table  of  Contents.  Add/delete  headings  as  necessary. 

First  order  headings  Helv  12  pt.  bold  (F6}.  cups, 

left  justified 

2  HRTs  before  (unless  first 
one,  then  6  HRTs) 

Second  order  headings  Helv  12  pt,  no  cros,  indent 

1 HRT  before 

Third  order  headings  Helv  12  pt,  no  caps,  indent, 

indent 

1  HRT  before 

Notes  Indent,  bold  (F6)  the  word  ’Note;’  or  'Notes:' 

If  info  is  especially  important,  then  bold  the  info  also. 

For  more  than  one  note,  number  the  notes.  See  CAL  User’s 
Guide  for  sample. 

Single/Multiples  Make  appropriate  change  (e.g.,  change  EXAMPLEXS  to 

example  or  EXAMPLES,  whichever  is  appropriate. 
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APPENDIX  G 

SAMPLE  TEMPLATE  FILE  FOR  COMPUTER-BASED  LESSON  DEVELOPMENT 


went  name:  info-1  (fi^Sm) 


$Segment  Title 

- - 


[20 


The  [1  ]  section  contains  the  information  that  is 

most  necessary  for  you  to  learn. 

Use  the[13]or[14  jarrow  keys  to  make  a  selection, 
th  jn  press  [4  ] . 


+ 

I 

I 

I 

+ 

I 


1 


- menu - 

SEGMENT  1  MENU 

$Title  of  Segmenr,  not  caps,  centered 


[  7] 

1. 

Introduction  and  Objective 

1 

1 

[  8] 

2  . 

Core  Information 

1 

[  9] 

3. 

More  Information  (delete, if  not  used) 

1 

[10] 

3  . 

Example  (delete,  if  not  used,  plural? 

) 

[11] 

4. 

Summary 

1 

1 

[12] 

5. 

Practice  Item  and  Feedback  (plural?) 

1 

1 

[5]=  Entered  this  menu  option  [6]=  Completed  this  menu  option 


F1=HELP  F2=EXIT  F3=GL0SSARV  F4=SEGMT  MENU  F5=MAIN  MENU  F10*SCOREBOARD 


DISPLAY  ATTRIBUTES: 
footnote  20;  grey/black:  "§m" 


line  color 
footnote  1 
footnote 
footnote 
footnote 
footnote 
footnote 
footnote 


bright_blue 

yellow7black  :  "Core  Information" 
oright_green/black  ;  "RETURN" 
yellow/black: 

II  i,  II 

$sio 
$sci 
$smi 


yellow/black: 
yellow/black: 
yellow/black: 
yellow/black: 

footnote  10:  yellow/black:  Sse 
footnote  11:  yellow/black:  $ss 
yellow/black:  $sp 
bright_green/black  : 
bright_green/black  : 
text  color:  bright_white 
menu  title  color:  yellow 
menu  title  background  color:  black 
menu  text  color:  bright_white 
menu  text  background  color:  black 
menu  selector  color:  yellow 
«nu  selector  line  number:  1 


footnote  12: 
footnote  13: 
footnote  14: 


"  up  " 
"down" 


KEY 


functional  area 
lesson  number 
segment  number 
emsn 

page  number,  number 


G-1 


Menu  border  color:  bright_blue 
\enu  background  color:  black 
screen  updaCe:  refresh 


CONTROL  ATTRIBUTES: 
branch  control 

terminating  condition:  cmsn  »»  1 

next  event:  e#5io  (intro  and  objective) 

activation  method:  branch 

branch  control 

terminating  condition:  cmsn  “  2 
next  event:  fiiSci  (core  info) 
activation  method:  branch 

branch  control 

terminating  condition:  cmsn  =  % 
next  event:  §#$mi(more  info) 
activation  method:  branch 

branch  control 

terminating  condition:  cmsn  =  % 
next  event:  @(t$e  (example) 
activation  method:  branch 

branch  control 

terminating  condition:  cmsn  =  % 
next  event:  @|$s  (summary) 
activation  method:  branch 

branch  control 

terminating  condition:  cmsn  =  % 
next  event:  §|$p  (practice) 
activation  method:  branch 

branch  control 

terminating  condition:  (ckr  ■  B)  or  (ckr  =  b) 
next  event:  pen 
activation  method:  branch 

branch  control 

terminating  condition:  ckr  =  <F1> 
next  event:  calfl  (help) 
activation  method:  call 

branch  control 

terminating  condition:  ckr  -  <F2> 
next  event:  calf 2  ( Exit/Save/Reenter } 
activation  method:  call 

branch  control 

terminating  condition:  ckr  *  <F3> 
next  event:  calf 3  (glossary) 
activation  method:  call 
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Ibranch  control 

terminating  condition:  ckr  •  <F1> 
next  event:  e#$m  {Components  menu) 
activation  method:  branch 

branch  control 

terminating  condition:  ckr  *  <F5> 
next  event:  ma inmenu  (main  menu) 
activation  method:  branch 

branch  control 

terminating  condition:  ckr  *  <F6> 
next  event:  calf6  (blank) 
activation  method:  call 

branch  control 

terminating  condition:  ckr  =  <F7> 
next  event:  calf 7  (blank) 
activation  method:  call 

branch  control 

terminating  condition:  ckr  =  <F8> 
next  event:  calfS  (blank) 
activation  method:  call 

branch  control 

terminating  condition:  ckr  =  <F9> 
next  event:  calf9  (blank) 
activation  method:  call 

branch  control 

terminating  condition:  ckr  =  <F10> 
next  event:  calflO  (scorecard) 
activation  method:  call 

MODIFICATION  LOG;  7/24/89  vel 
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APPENDIX  H 

SAMPLE  REVIEW/REVISION  CYCLE  PLAN 
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SAMPLE  REVIEU/REVISION  CTCLE  PLAN 


1.  OcvelepMnt  Instructional  Developers  develop  the  first  draft  (Draft  *1)  of  a  lesson  using  the 
of  Draft  SEID  process. 

#1  by 

Instructional 

Developers 

2.  Internal  Review  the  lesson  materials  in-house  before  sending  a  first  draft  to  SMEs. 

tevieu/  Use  a  standardized  review  form  throughout  the  entire  review/ revision  process. 

Revision  of 

Draft  f1  by 

Instructional  A.  Uho  Reviews.  The  lesson  author  should  review  the  draft  first.  Then  at  least 
Developers  one  other  instructional  developer  should  review  the  draft  to  provide  a  fresh 

viewpoint. 


B.  Uhat  to  Review  For.  Write  standardized  instructions  for  the  review.  For 
example,  develop  a  checklist  of  specific  instructions  for  the  review  process 
(indicating  when  the  review  must  be  conpleted),  as  well  as  uhat  to  look  for 
when  reviewing  materials  (rather  than  saying  "Just  note  anything  you'd  like 
to  be  better").  Review  the  materials  for: 

*  Content  Validity.  Although  the  SHE  is  the  expert  on  content  validity,  the 
lesson  author  can  review  for  obvious  content  flaws  based  on  the  materials 
s/he  has  been  using  to  write  the  draft.  The  second  reviewer  should 
review  for  readability,  clarity,  interest,  arwJ  effectiveness  of  examples/ 
practice  items. 


Spelling,  Graanatical  Errors.  Alt  instructional  developers  can  review  for 
spelling  and  grammatical  errors.  The  more  of  these  types  of  errors  that  can  be 
flushed  out  during  internal  review,  the  more  the  SHE  reviews  can  concentrate  on 
content  validity. 


Format  Issues.  Although  the  lesson  format  should  have  been  planned  during  the 
design  phase  of  the  SEID  process,  actual  development  of  the  materials  may  reveal 
deficiencies  in  the  original  plan.  These  deficiencies  can  be  noted  and 
suggestions  for  corrections  can  be  made  during  this  review  stage. 


After  the  internal  review  is  complete  and  the  revisions  have  been  made  in-house,  send 
Draft  #1  to  the  SME<s). 
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3.  SME  Review/  |a.  Uho  Reviews.  If  more  than  one  SHE  needs  to  review  the  material,  plan  the  sequence 

Revision  of  |  of  which  SME  receives  the  material  and  on  what  date.  As  recommended  elsewhere,  make 

Draft  #1  j  sure  that  the  last  SME/coordinator  to  review  the  materials  is  the  SHE/coordinator 

l  with  the  final  approval  authority. 


|B.  Uhat  to  Review  For.  SMEs  must  focus  on  reviewing  the  validity  of  the  content, 

I  structure,  and  presentation  sequence  of  the  instructional  materials.  However,  SMEs 

should  also  review  for  the  same  items  assessed  during  internal  review  (e.g., 
spelling,  grammatical  errors,  interest,  clarity,  readability,  etc.).  Encourage 
SMEs  to  be  as  thorough  as  possible,  especially  in  the  review  of  the  first  draft. 
Provide  and  instruct  SMEs  to  use  the  standardized  review  form  and  the  checklist  of 
the  review/ revision  procedure. 


After  the  SMEls)  have  reviewed  the  draft,  the  review  form  should  be  returned  to  the 
lesson  author  for  revision. 


4.  Internal 
Review/ 
Revision  of 
Draft  #2 

After  making  the  revisions,  you  might  want  to  conduct  a  second  internal  review  to  make 
sure  all  the  SMEs'  revisions  have  been  made,  and  that  there  are  no  other  errors.  Again, 
the  lesson  author  and  at  least  one  other  team  mentier  should  review  the  materials. 

5.  SME  Review/ 
Revision  of 
Draft  #2,  if 
appropriate 

A  second  SME  review/revision  may  be  needed  if  SMEs  suggested  major  or  nianerous  changes 
to  the  first  draft.  If  this  is  the  case,  send  SMEs  another  standardized  review  form 
along  with  the  second  draft.  If  only  part  of  the  lesson  was  extensively  revised,  you 
might  just  return  that  part  to  the  SME  for  a  secord  review. 

APPENDIX  I 

SAMPLE  STANDARDIZED  REVIEW/REVISION  FORM 


I-O 


Pm  H  K  H  Q 


DEVELOPER’S  CONTENT  FEEDBACK 


CAL 

SCREEN 


COMMENTS 


APPENDIX  J 

SAMPLE  COORDINATOR  INSTRUCTIONS  FOR  FORMATIVE  EVALUATION 


J-0 


EVALUATION  COORDINATOR  WORKSHEET 

(To  be  used  for  each  participant) 


Date  _ 

Evaluation  Coordinator  Organization 

PARTICIPAMT  INFORMATION: 

Organization  _ 

Name  _ 

Identification  number*  _ 

Phone  number  _ 


Print  version?  _  Computer  version?  _ 

Location  of  computer  _ 

Is  the  computer  a  Zenith  248?  See  Users  Guide. 

Is  the  computer  set  up  and  loaded  (by  you  or  by  one  of  the 
participants)?  Yes  _  No  _ 

Is  the  schedule  for  computer  usage  time  posted? 


Date  for  participant  to  begin  instruction 

Supervisor  who  has  given  approval  _ 

Time  per  week  available  for  training  _ 

Facilitator  for  this  participant  _ 

Facilitator  hours  available  to  participant 


♦Please  assign  each  participant  a  unique  identification  number  and 
make  sure  that  this  number  appears  on  all  materials.  For  example, 
FEOl,  FE02,  etc.)  Please  also  let  the  participant  know  what  this 
number  is,  so  all  materials  can  be  identified. 
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EVALUATION  COORDINATOR  INFORMATION 


The  goal  of  the  formative  evaluation  is  to  obtain  feedback  from 
participants  on  questions  such  as: 

—  do  the  participants  understand  the  content? 
do  the  lessons  flow  well? 

how  long  does  it  take  each  participant  to  accomplish  each  lesson? 

A  feedback  form  packet  will  be  provided  to  each  participant  to  gather; 
1)  information  on  some  key  background  questions  and  2)  their  ideas  on 
format  and  content  questions  to  improve  the  materials.  The  evaluation 
coordinator  will  complete  a  worksheet  for  each  participant  to  help 
organize  the  administration  of  the  lessons. 

As  a  reminder,  please  do  not  alter  any  of  the  training  materials. 
Present  them  to  the  participants  just  as  they  are  given  to  you. 

For  the  (DATE)  evaluation,  it  would  be  best  to  have  at  least  five  but 
no  more  than  ten  participants  complete  each  print  version  of  the 
lessons  and  the  same  number  of  participants  complete  each  computer- 
based  version. 


Facilitator  help 

Each  participant  must  have  someone  who  can  serve  as  a  facilitator  to 
help  with  the  lesson  content,  someone  they  can  call  with  questions  or 
clarifications.  Facilitators  should  have  completed  the  (SUBJECT  AREA 
COURSE)  or  its  equivalent  or  should  have  had  at  least  (AMOUNT)  of 
experience  in  (SUBJECT  AREA) . 

The  facilitator  would  be  the  first  person  to  whom  the  participant 
would  direct  questions.  If  more  information  is  needed,  the 
facilitator  can  direct  (or  have  participants  do  so)  questions  to  the 
Evaluation  Coordinator.  The  Evaluation  Coordinator  can  contact  (YOUR 
ORGANIZATION'S  NAME  AND  POINTS  OF  CONTACT  WITH  PHONE  NUMBERS). 


Computer  help  for  the  computer-based  version 

— To  load  the  training  materials,  refer  to  the  Users  Guide. 
— For  assistance,  contact  your  local  computer  specialists. 
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TASKS  FOR  EVALUATION  COORDINATORS 


Obtain  or  praoare  sufflcl«nt  aaf  rl»ls 

EACH  participant  will  need: 

1  Evaluation  Coordinator  Worksheet  per  lesson  (participant 

information,  to  be  filled  out  by  the  evaluation  coordinator) 
1  Participant  Feedback  Form  packet  per  lesson  (participant 

feedback  from  lessons,  to  be  filled  out  by  the  participant) 

Each  PRINT  participant  will  ALSO  need  1  Lesson  Booklet  per  lesson 

Each  COMPUTER  ASSISTED  LEARNING  participant  will  need: 

—  1  Users  Guide  per  lesson, 

—  A  computer  loaded  with  the  training  materials,  OR  a  set  of 
floppy  disks  so  they  can  load  the  materials  before  beginning, 
—  A  schedule  when  this  computer  is  available  for  his/her  use, 

—  1  Student  Disk  for  EACH  lesson  to  be  done  (have  the 

participants  label  their  disks  with  their  ID  numbers  and  the 
lesson  numbers) . 


It  would  be  helpful  to  label  each  set  of  lesson  materials  with  an 
identification  number  unique  to  each  participant  (and  let  the 
participant  know  his/her  identification  number) ,  so  all  materials  can 
be  easily  identified. 

Since  the  lessons  need  to  be  completed  and  delivered  to  the  evaluation 
coordinator  by  (DATE) ,  some  way  of  checking  on  participant  progress 
might  be  necessary. 


Collection  of  materials 

After  the  participants  have  completed  the  lessons,  the  materials  will 
be  collected  and  mailed  to  (LOCATION) .  Materials  to  return  include: 

—  hardcopy  lessons, 

—  student  disks, 

—  feedback  form  packets, 

—  evaluation  coordinator  worksheets 

—  and  any  other  materials  on  which  participants  have  written  comments 

Please  mail  completed  materials  as  they  are  returned  to  you.  You  may 
use  the  mailing  labels  supplied  to  you.  We  suggest  that  you  send  a 
packet  of  completed  materials  at  the  end  of  each  week.  THE  FINAL 
DEADLINE  FOR  ALL  MATERIALS  to  be  returned  to  the  evaluation 
coordinator  is  (DATE) . 
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APPENDIX  K 

SAMPLE  EVALUATOR  FORM  FOR  FORMATIVE  EVALUATION 


K-0 


YOUR  ORGANIZATION  OR  PROJECT  TITLE 


PARTICIPANT  FEEDBACK  -  DRAFT  LESSON  TRYOUTS 

We  are  exploring  the  conversion  of  parts  of  the  [Course  Name]  to  a 
self-study  option.  We  are  evaluating  draft  versions  of  the  first 
lessons  converted  to  this  self-study  format.  We  are  not  testing  your 
knowledge  of  the  subject  matter,  but  rather  evaluating  how  well  the 
materials  actually  teach.  The  feedback  you  provide  on  these  lessons 
will  help  us  improve  the  instruction  not  only  for  these  lessons,  but 
also  for  other  lessons  in  this  field. 

1.  Name  or  ID  Number  (ask  your  evaluation  coordinator) 


2.  Job  title  _ 

3.  Work  address: 


4  .  Vork  phone :  _ ^ _ 

5.  Job  Supeirvisor  Name: _ 

6.  Please  complete  the  following  background  information. 

Military  or  Civilian  Grade  _ 

Specialty  or  Occupational  Code  _ 

Experience  in  (TOPIC  AREA  1)  _  years 

Experience  in  a  (TOPIC  AREA  2)  _  years 

Total  Military  or  Federal  Service  _  years 

Highest  Academic  Degree  _ 

Military  Schools  (C/.E) _ (SSC) _ 

Last  or  Current  job  Assignment  _ 

Projected  Nex^'  Assignment _ 

P.  Have  you  been  a  (SCHOOL  X)  student  in  the  past?  _  Yes  _  No 

If  YES,  what  class  were  yo»’  in? _ 
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GENERAL  INSTRUCTIONS  FOR  RLUNG  OUT  THE  FEEDBACK  FORMS: 


The  draft  materials  you  have  been  given  contain  a  pre-quiz,  a  lesson 
consisting  of  several  segments  of  instruction,  and  a  post-quiz.  For 
each  quiz  and  segment,  there  is  a  feedback  form. 

1.  As  you  work  on  each  component  of  the  lesson,  please  note  the  time 
it  takes  you  in  the  spaces  indicated  on  these  feedback  forms.  If  you 
are  interrupted,  please  estimate  how  long  you  actually  spend  on  the 
quiz  or  segment. 

2.  As  you  proceed  through  the  materials,  please  write  any  comments, 
questions,  or  suggestions  directly  on  the  pages  of  the  lesson  booklet 
or  (if  you  are  taking  the  computer-based  version)  on  the  Computer 
Screen  Feedback  sheets  at  the  back  of  this  packet. 

3.  As  you  complete  each  component  of  the  lesson,  please  fill  out  the 
feedback  form  for  that  particular  component.  Be  as  specific  as 
possible  in  making  comments  and  suggestions. 

4.  At  the  end,  please  fill  out  the  Overall  Lesson  Feedback  Form. 

5.  If  you  have  any  problems  or  questions  about  the  materials, 
contact  your  facilitator. 

6.  If  you  have  any  problems  or  questions  about  time  available  for 
training,  first  contact  your  supervisor.  If  he  or  she  cannot  resolve 
the  problem,  please  contact  your  evaluation  coordinator. 

7.  AS  you  complete  each  lesson,  please  return  the  lesson  and  all  of 
its  associated  materials  to  the  evaluation  coordinator.  THE  FINAL 
DEADLINE  FOR  ALL  MATERIALS  IS  (DATE) . 


Thank  you  for  your  help! 


Now,  you  are  ready  to  begin  the  lesson.  When  you  have  completed  the 
pre-quiz,  return  to  the  next  page  of  this  feedback  form. 
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SEGMENT  1  FEEDBACK  FORM 


Please  circle  1,  2,  or  3  and  write  comments  and  suggestions  for  each 
question  If  you  need  more  space,  please  write  on  the  back  of  this  sheet. 

Please  note  your  Total  time  _  for  completing  Segment  1. 

YES  SOMEWHAT  NO 

1.  Did  you  understand  from  the  Lesson  Objectives  123 

what  you  would  be  able  to  do  after  completing  the 
segment?  If  not,  please  suggest  how  the 
objectives  could  be  rephrased. 


2.  Do  you  now  feel  you  can  perform  the  activities  123 

stated  in  the  Lesson  Objectives?  If  not,  why  not? 


3 .  Was  the  Core  Information  presented  clearly?  123 

If  not,  how  could  it  be  clearer? 


4.  If  there  was  a  More  Information  section,  did  it  1  2  3 

add  important  information?  If  not,  what  should  be 
added  or  deleted? 


5.  If  there  were  Examples .  were  they  clear?  123 

If  you  can  think  of  any  better  examples,  please 
write  them  here  or  on  the  back  of  this  sheet. 


6.  Were  there  any  areas  that  could  benefit  from  123 

added  or  deleted  graphics?  If  so,  please  advise. 


7.  Were  the  Practice  Items  clearly  stated?  123 

If  not,  how  could  they  be  stated  more  clearly? 
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8. 


Would  more  Practice  Items  be  helpful? 
If  yes,  what  topics  should  they  cover? 


YES  SOMEITHAT  NO 

12  3 


9.  Did  the  Practice  Feedback  adequately  explain  123 

why  your  answers  were  correct/ incorrect? 

If  not,  how  would  you  rephrase  it  to  provide 
better  feedback? 


10.  Did  you  have  any  unanswered  questions  during  123 

the  segment?  If  yes,  what  were  they? 


Please  circle  a  number  for  each  item  that  you  feel  best  describes  the 
segment.  Then  write  comments  or  suggestions  for  each  item. 

1.  DIFFICULT  TO  FOLLOW  EASY  TO  FOLLOW 

1 - 2 - 3 - 4 - 5 - 6 - 7 - 8 - 9 - 10 

Comments ; 


2.  BENEFICIAL  WASTE  OF  TIME 

1 - 2 - 3 - 4 - 5 - 6 - 7 - 8 - 9 - 10 

Comments : 


3.  BORING  INTERESTING 

1 - 2 - 3 - 4 - 5 - 6 - 7 - 8 - 9 - 10 

Comments : 


4.  TOO  DETAILED  NOT  ENOUGH  DETAIL 

1 - 2 - 3 - 4 - 5 - 6 - 7 - 8 - 9 - 10 

Comments : 


^Jow  go  on  to  the  next  segment  in  your  lesson  booklet  or  on  the  computer. 
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COMPUTER-BASED  LEARNING  FEEDBACK  FORM 

1-  Were  you  able  to  install  the  lesson  without  assistance? 

_  Yes  _  No  _  It  was  already  installed 

If  NO,  what  assistance  was  necessary? 

COMMENTS/ SUGGESTIONS : 


2.  Were  you  able  to  use  the  operating  procedures  without  help? 

_  Yes  _  No 

COMMENTS/SUGGESTIONS : 


Did  vou  use  the  function  keys? 

Yes 

_  No 

If  YES,  which  ones?  (Please  circle) 

Were 

^hey  useful? 

YES 

SOMEWHAT 

NO 

FI  -  Help 

1 

2 

3 

F2  -  Exit/Save/Reenter 

1 

2 

3 

F3  -  Glossary 

1 

2 

3 

F4  -  Lesson  Components 

1 

2 

3 

F5  -  Main  Menu 

1 

2 

3 

FIO  -  Scoreboard 

COMMENTS/SUGGESTIONS : 

1 

2 

3 

K-5 


DISTRIBUTION  LIST 


Chief  of  Naval  Operations  (OP-01B2) 

Defense  Technical  Information  Center  (DTIC)  (2) 


